Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 08:53
    veikkoeeva commented #6257
  • 08:53
    veikkoeeva commented #6257
  • 08:52
    veikkoeeva commented #6257
  • 08:51
    veikkoeeva commented #6257
  • Jan 26 22:28
  • Jan 26 20:45
    eiadxp starred dotnet/orleans
  • Jan 26 20:17
    Anthonywolfe starred dotnet/orleans
  • Jan 26 05:15
    sergeybykov closed #6135
  • Jan 26 05:15
    sergeybykov closed #6123
  • Jan 26 05:14
    sergeybykov assigned #6113
  • Jan 26 05:14
    sergeybykov unassigned #6113
  • Jan 26 05:01
    sergeybykov closed #6038
  • Jan 26 05:01
    sergeybykov commented #6038
  • Jan 26 05:00
    sergeybykov commented #6033
  • Jan 26 04:44
    sergeybykov closed #5968
  • Jan 26 04:44
    sergeybykov commented #5968
  • Jan 26 04:42
    sergeybykov closed #5965
  • Jan 26 04:31
    sergeybykov milestoned #6256
  • Jan 26 04:31
    sergeybykov assigned #6256
  • Jan 26 04:29

    sergeybykov on master

    Fix codegen incremental rebuild… (compare)

David Christensen
@OracPrime
I've got my two silos up and running, and I ought to have entirely independent runs in each silo - each grain just calling to other grains in the same silo. However it's 8-10 times as slow as single silo, so I'm guessing I'm getting unexpected cross-silo calls. How can I find out what these are? Are there any hooks or logs I can use?
Reuben Bond
@ReubenBond
@OracPrime you could look at the silo-to-silo message counts which are dumped as a part of statistics to see how many cross-silo messages are being sent
Those send & receive stats are emitted per remote silo
Jorge Candeias
@JorgeCandeias
@OracPrime By the way that's normal behaviour. Orleans places regular grains at random by default. You can change this behaviour by picking some other placement strategy for the grain. For example, if you tag a grain implementation with [PreferLocalPlacement], Orleans will attempt to activate each instance in the same silo of the call that triggers the activation. This can skew the grain distribution across the cluster but can help with groups that should stay together.
Reuben Bond
@ReubenBond
You could also use log messages with or without a grain call filter to emit details about which grains are calling which other grains and which silo each grain lives on. Then you could determine the full set of inter-grain calls and work out if the distribution seems right
David Christensen
@OracPrime
@JorgeCandeias I have a custom placement which appeared, I thought, to correctly partition my grains between silos. And each grain should only talk to other grains in the same silo. I'm doing a calculation which I have sliced, and the slices are self-contained until the very end
You see, apart from client->silo calls, I'm expecting precisely one inter-silo call, when a ModelParms object is interrogated by a ModelParmsCache stateless worker in the other silo. But I can't imagine one call is taking 10s to 80s :(
Jorge Candeias
@JorgeCandeias
@OracPrime 80s? That's odd. Orleans early timeouts the call at 30s with an exception.
David Christensen
@OracPrime
@JorgeCandeias it's not one call, it's ten slices with ten sub-slices. The client calls for each 100 sub slices, but they should locate in silos according to top level slice number
Reuben Bond
@ReubenBond
Let's examine assumptions, etc, so we can see why you're seeing this behavior
David Christensen
@OracPrime
Is there a call to trigger a dump of the Stats I can do when it finishes running?
Reuben Bond
@ReubenBond
Not yet (feel free to PR), but you can modify the logging interval via StatisticsOptions.LogWriteInterval
Jorge Candeias
@JorgeCandeias
Is this code something you could share in an issue?
David Christensen
@OracPrime
@JorgeCandeias not really :( . I've done a quick check and the top level calcs (so the ones that the client calls in to) are definitiely being done in the right silos. I'll check the ancestors...
Reuben Bond
@ReubenBond
Issues or DMs are a good alternative
David Christensen
@OracPrime
I've turned on my own logging inside the grains, dumping the key which should determine which silo it is in, and everything seems to be happening in the correct silos. So why so slow?
Reuben Bond
@ReubenBond
How many unique grains are you creating in what period of time? Maybe the directory is getting saturated?
David Christensen
@OracPrime
A couple of thousand in a few seconds
Reuben Bond
@ReubenBond
The directory could be part of it. I assume they are very short-lived, essentially single-use grains. In likelihood, they could be merged into fewer grains.
makuchma
@makuchma

@Vlad-Stryapko

@makuchma
Have you considered something like

var e1 = GetItems() //task
var e2 = g.GetIems //task
await Task.WhenAll(e1, e2)
//do whatever you want with result

Thanks, good suggestion.

Reuben Bond
@ReubenBond
btw, it's better to troubleshoot these kinds of things over DM or GH issues, @OracPrime
David Christensen
@OracPrime
OK
Reuben Bond
@ReubenBond
Otherwise the firehose of messages drowns people out
makuchma
@makuchma
The doc states we can use persistence with CosmosDB via Microsoft.Orleans.Persistence.AzureStorage. However, there is a similar package in contrib: https://github.com/OrleansContrib/Orleans.CosmosDB. What's the difference?
Kyle Dodson
@seniorquico
@makuchma The Microsoft.Orleans.Persistence.AzureStorage library makes use of Blobs, Queues, and Tables. CosmosDB provides a compatible Table API:
https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction
iirc the Orleans.CosmosDB library uses the Cosmos API, not the Table API.
makuchma
@makuchma
@seniorquico Thanks, that makes sense. I'll have to read up on CosmosDB using Cosmos API vs Table API to better understand the trade-offs, I'm not sure why I should pick one or the other.
Kyle Dodson
@seniorquico
i think using Cosmos with the Table API is just for compatibility... you get the increased perf and other Cosmos benefits, without needing to integrate with a different API.
but, in our experience, Cosmos costs significantly more than using the plain old Table Storage.
Sergey Bykov
@sergeybykov
@richorama or @galvesribeiro, could you please merge my OrleansContrib/meetups#18 and OrleansContrib/meetups#19. Thanks!
Gutemberg Ribeiro
@galvesribeiro
done! Thank you @sergeybykov :smile:
Sergey Bykov
@sergeybykov
Thanks!
@galvesribeiro While at it, can you add an empty line after the presentation link?
The font is too big otherwise.
image.png
Gutemberg Ribeiro
@galvesribeiro
how do I add an empty line on markdown? just <br/> ?
Sergey Bykov
@sergeybykov
No-no, just an empty line in the text.
Gutemberg Ribeiro
@galvesribeiro
image.png
that :)
Sergey Bykov
@sergeybykov
👍
Gutemberg Ribeiro
@galvesribeiro
done
Sergey Bykov
@sergeybykov
Thanks man.
Gutemberg Ribeiro
@galvesribeiro
:+1:
Veikko Eeva
@veikkoeeva
:+1: Add my drive-by-thumbs too. :)
I'll comment https://github.com/dotnet/orleans/issues/6257#issuecomment-577722185 tomorrow. Thanks for merging my notes one OneBoxDeployment. It may not be apparent, but if one looks at securities, i.e. cash-flow from the future and selling debt, it makes a lot of sense. One example being multi-modal transport projects and building infrastructure and doing something smarter than toll booths to generate perceived benefits. I think Orleans could play a part here if Microsoft doesn't "pull off Oracles" with licenses. :)
Eugene Krapivin
@EugeneKrapivin
@sergeybykov Thanks for your answer. I think I was not focused enough in the first question and had a mistake in the second.
a. Given a stream of events in kafka would it be possible to open a grain per-event in said stream in order or distribute the processing load?
the only method of doing such a thing I see is to bootstrap a worker grain to begin consuming from a kafka consumer (not an Orleans stream since the stream doesn't originate within the Orleans cluster) and then to redistribute the load in the Orleans cluster by grains per event. I haven't found a sound way of doing it "the Orleans way"
b. I asked the wrong thing in my second question, the correct question should be: is it possible to facilitate Orleans streams between Orleans clusters?
In Gigya we are using Orleans as a micro-service platform in which each service is a standalone Orleans cluster and communication between services is facilitated by HTTP requests
@yevhen Its not a question of performance more a question of my ability to introduce a new framework in an enterprise solution :)
Samir Mowade
@sammym1982

@sammym1982 What does the reminder handler code look like?

Sorry for late response on this. I was not clear on my original description of test. I wanted to check how far orleans can stretch for reminders which fires within 1 minute window. When there are around ~50k reminders across 25k grains which needs to be fired at interval of 1 min interval that's when I start seeing these errors. I understand this mostly because the reminders are starving for getting served but I wanted to understand what casuing this and can these be ignored as it will get served or will it wait for next tick. A reminder for a given grain always fires every 1 min. I dont see any CPU or memory pressure. And Reminder handler is just doing log message.

This is certainly is more that what I need but knowing the cause will help prepare keep mitigations in place if some burst of reminders gets registered like this. Thanks.

 public Task ReceiveReminder(string reminderName, TickStatus status)
        {
            logger.LogWarning(12345, $"{this.GetPrimaryKeyLong()} Reminder fired for {reminderName} @ {DateTime.Now}");
            return Task.CompletedTask;
        }

Some warnings

info: Orleans.Runtime.Catalog[100520]
      Tried to create a duplicate activation S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, but we'll use S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba instead. GrainInstanceType is HelloWorld.Grains.HelloGrain. Primary Directory partition for this grain is S127.0.0.1:11111:317674027. Full activation address is [ActivationAddress: S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, Full GrainId: [GrainId: *grn/F057335D/000000000000000000000000000012e303fffffff057335d-0x36D36D96, IdCategory: Grain, BaseTypeCode: -262720675 (xF057335D), PrimaryKey: 4835 (x12E3), UniformHashCode: 919825814 (0x36D36D96)], Full ActivationId: @4769cbac85c8d09bb8f82367c231e19a0000000000000000]. We have 1 messages to forward.
info: Orleans.Runtime.Dispatcher[101033]
      Forwarding 1 requests destined for address S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a to address S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba after Duplicate activation.
info: Orleans.Runtime.Dispatcher[101031]
      Trying to forward after Duplicate activation, ForwardCount = 0. OldAddress = S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, ForwardingAddress = S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba, Message NewPlacement Request S127.0.0.1:11111:317674027*stg/0/00000000@S00000000->S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a #900819: , Exception: (null).

Error

fail: Orleans.Runtime.ReminderService.LocalReminderService[102915]
      Could not deliver reminder tick for [Reminder_2, GrainReference:*grn/F057335D/0000000000000000000000000000234103fffffff057335d-0x08E0A882, 00:01:00, 2020-01-25 23:52:36.369 GMT, 0, 18112, Ticking], next 1/26/2020 2:21:24 AM.
System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request S127.0.0.1:11111:317674027*stg/0/00000000@S00000000->S127.0.0.1:11111:317674027*grn/F057335D/00002341@7a3179b2 #6115446: . Target History is: <S127.0.0.1:11111:317674027:*grn/F057335D/00002341:@7a3179b2>.
   at Orleans.Runtime.ReminderService.LocalReminderService.LocalReminderData.OnTimerTick(AverageTimeSpanStatistic tardinessStat, ILogger Logger) in D:\build\agent\_work\17\s\src\Orleans.Runtime\ReminderService\LocalReminderService.cs:line 584
@JorgeCandeias
Jorge Candeias
@JorgeCandeias
@sammym1982 Can't see anything wrong from that sample, apart from synchronous console logging slowing things down. Would it be okay for you to share the test project on a issue?