I've got my two silos up and running, and I ought to have entirely independent runs in each silo - each grain just calling to other grains in the same silo. However it's 8-10 times as slow as single silo, so I'm guessing I'm getting unexpected cross-silo calls. How can I find out what these are? Are there any hooks or logs I can use?
Those send & receive stats are emitted per remote silo
@OracPrime By the way that's normal behaviour. Orleans places regular grains at random by default. You can change this behaviour by picking some other placement strategy for the grain. For example, if you tag a grain implementation with
[PreferLocalPlacement], Orleans will attempt to activate each instance in the same silo of the call that triggers the activation. This can skew the grain distribution across the cluster but can help with groups that should stay together.
@JorgeCandeias I have a custom placement which appeared, I thought, to correctly partition my grains between silos. And each grain should only talk to other grains in the same silo. I'm doing a calculation which I have sliced, and the slices are self-contained until the very end
You see, apart from client->silo calls, I'm expecting precisely one inter-silo call, when a ModelParms object is interrogated by a ModelParmsCache stateless worker in the other silo. But I can't imagine one call is taking 10s to 80s :(
The doc states we can use persistence with CosmosDB via Microsoft.Orleans.Persistence.AzureStorage. However, there is a similar package in contrib: https://github.com/OrleansContrib/Orleans.CosmosDB. What's the difference?
@makuchma The
https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction
Microsoft.Orleans.Persistence.AzureStorage library makes use of Blobs, Queues, and Tables. CosmosDB provides a compatible Table API:https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction
iirc the
Orleans.CosmosDB library uses the Cosmos API, not the Table API.
but, in our experience, Cosmos costs significantly more than using the plain old Table Storage.
@richorama or @galvesribeiro, could you please merge my OrleansContrib/meetups#18 and OrleansContrib/meetups#19. Thanks!
@galvesribeiro While at it, can you add an empty line after the presentation link?
The font is too big otherwise.
that :)
I'll comment https://github.com/dotnet/orleans/issues/6257#issuecomment-577722185 tomorrow. Thanks for merging my notes one OneBoxDeployment. It may not be apparent, but if one looks at securities, i.e. cash-flow from the future and selling debt, it makes a lot of sense. One example being multi-modal transport projects and building infrastructure and doing something smarter than toll booths to generate perceived benefits. I think Orleans could play a part here if Microsoft doesn't "pull off Oracles" with licenses. :)
@sergeybykov Thanks for your answer. I think I was not focused enough in the first question and had a mistake in the second.
a. Given a stream of events in kafka would it be possible to open a grain per-event in said stream in order or distribute the processing load?
the only method of doing such a thing I see is to bootstrap a worker grain to begin consuming from a kafka consumer (not an Orleans stream since the stream doesn't originate within the Orleans cluster) and then to redistribute the load in the Orleans cluster by grains per event. I haven't found a sound way of doing it "the Orleans way"
b. I asked the wrong thing in my second question, the correct question should be: is it possible to facilitate Orleans streams between Orleans clusters?
In Gigya we are using Orleans as a micro-service platform in which each service is a standalone Orleans cluster and communication between services is facilitated by HTTP requests
a. Given a stream of events in kafka would it be possible to open a grain per-event in said stream in order or distribute the processing load?
the only method of doing such a thing I see is to bootstrap a worker grain to begin consuming from a kafka consumer (not an Orleans stream since the stream doesn't originate within the Orleans cluster) and then to redistribute the load in the Orleans cluster by grains per event. I haven't found a sound way of doing it "the Orleans way"
b. I asked the wrong thing in my second question, the correct question should be: is it possible to facilitate Orleans streams between Orleans clusters?
In Gigya we are using Orleans as a micro-service platform in which each service is a standalone Orleans cluster and communication between services is facilitated by HTTP requests
@yevhen Its not a question of performance more a question of my ability to introduce a new framework in an enterprise solution :)
@sammym1982 What does the reminder handler code look like?
Sorry for late response on this. I was not clear on my original description of test. I wanted to check how far orleans can stretch for reminders which fires within 1 minute window. When there are around ~50k reminders across 25k grains which needs to be fired at interval of 1 min interval that's when I start seeing these errors. I understand this mostly because the reminders are starving for getting served but I wanted to understand what casuing this and can these be ignored as it will get served or will it wait for next tick. A reminder for a given grain always fires every 1 min. I dont see any CPU or memory pressure. And Reminder handler is just doing log message.
This is certainly is more that what I need but knowing the cause will help prepare keep mitigations in place if some burst of reminders gets registered like this. Thanks.
public Task ReceiveReminder(string reminderName, TickStatus status)
{
logger.LogWarning(12345, $"{this.GetPrimaryKeyLong()} Reminder fired for {reminderName} @ {DateTime.Now}");
return Task.CompletedTask;
}Some warnings
info: Orleans.Runtime.Catalog[100520]
Tried to create a duplicate activation S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, but we'll use S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba instead. GrainInstanceType is HelloWorld.Grains.HelloGrain. Primary Directory partition for this grain is S127.0.0.1:11111:317674027. Full activation address is [ActivationAddress: S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, Full GrainId: [GrainId: *grn/F057335D/000000000000000000000000000012e303fffffff057335d-0x36D36D96, IdCategory: Grain, BaseTypeCode: -262720675 (xF057335D), PrimaryKey: 4835 (x12E3), UniformHashCode: 919825814 (0x36D36D96)], Full ActivationId: @4769cbac85c8d09bb8f82367c231e19a0000000000000000]. We have 1 messages to forward.
info: Orleans.Runtime.Dispatcher[101033]
Forwarding 1 requests destined for address S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a to address S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba after Duplicate activation.
info: Orleans.Runtime.Dispatcher[101031]
Trying to forward after Duplicate activation, ForwardCount = 0. OldAddress = S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a, ForwardingAddress = S127.0.0.1:11111:317674027*grn/F057335D/000012e3@8f33b6ba, Message NewPlacement Request S127.0.0.1:11111:317674027*stg/0/00000000@S00000000->S127.0.0.1:11111:317674027*grn/F057335D/000012e3@c231e19a #900819: , Exception: (null).Error
fail: Orleans.Runtime.ReminderService.LocalReminderService[102915]
Could not deliver reminder tick for [Reminder_2, GrainReference:*grn/F057335D/0000000000000000000000000000234103fffffff057335d-0x08E0A882, 00:01:00, 2020-01-25 23:52:36.369 GMT, 0, 18112, Ticking], next 1/26/2020 2:21:24 AM.
System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request S127.0.0.1:11111:317674027*stg/0/00000000@S00000000->S127.0.0.1:11111:317674027*grn/F057335D/00002341@7a3179b2 #6115446: . Target History is: <S127.0.0.1:11111:317674027:*grn/F057335D/00002341:@7a3179b2>.
at Orleans.Runtime.ReminderService.LocalReminderService.LocalReminderData.OnTimerTick(AverageTimeSpanStatistic tardinessStat, ILogger Logger) in D:\build\agent\_work\17\s\src\Orleans.Runtime\ReminderService\LocalReminderService.cs:line 584
@JorgeCandeias