dotnet/orleans

Orleans - Distributed Actor Model

Sergei Namniasov

@tesh42

Am I right, that when I send a request to a grain and the grain is not responding for awhile (because of connection issues or there is some CPU usage spike), I have to wait till the request times out before I can send another one (which could be routed to the same server)? Is it possible to send a request to the grain on one server first and then send it to same grain on another server shortly after that to reduce tail latency in case the first server is overloaded or has network problems?

Roger Creyke

@creyke

What is the most elegant way to tell a grain instance activation to deactivate?

I appreciate this is an anti-pattern.

Veikko Eeva

@veikkoeeva

Too good to not be pasted here: https://gvnshtn.com/maersk-me-notpetya/ . :)

Reuben Bond

@ReubenBond

@creyke DeactivateOnIdle? or is it a remote grain?

Roger Creyke

@creyke

that's the one thanks

michaeltdaniels

@michaeltdaniels

@pawanrao - based on the below, it looks like the GrainTimer catches and ignores all exceptions.
If I'm reading that right, InconsistentStateException thrown from a timer callback would never cause deactivation.
https://github.com/dotnet/orleans/blob/bb06f41d19ac188be9d014e63850e4eab4df871e/src/Orleans.Runtime/Timers/GrainTimer.cs#L108

michaeltdaniels

@michaeltdaniels

Are timer caveats documented somewhere?

There seems to be a few gotchya's like this (others being reentrancy, not going through grain interceptors) compared to other flows of grain code execution (grain calls / stream callbacks / reminder receives).

dotnet/orleans#2574 relates to the reentrancy issue.

michaeltdaniels

@michaeltdaniels

@pawanrao - just found this dotnet/orleans#4646 which relates to a timer tick throwing InconsistentStateException not triggering deactivation

Elan Hasson

@ElanHasson

Hey Folks!

I am looking for a platform to implement our application on that provides "managed" event-sourcing and all the complexities that come with that. I believe these are JournaledGrains in Orleans.

My intention is to model Aggregates (in the DDD sense) using JournaledGrains and use notifications to publish the domain events to our RabbitMQ broker so other domains can get at them (and other non-orleans services within our domain)

Our current architecture works based on a competing event consumer model and comes with lots of extra complexity that comes with unordered event delivery and a-least-once message delivery.

Elan Hasson

@ElanHasson

I believe the turn-based scheduling model in Orleans would "remove" that from our plate.

pawanrao

@pawanrao

thanks @michaeltdaniels
I think that answers my question

Elan Hasson

@ElanHasson

Is my assessment correct?

Veikko Eeva

@veikkoeeva

https://twitter.com/xoofx/status/1275878129786109954 ha haa, /cc @JorgeCandeias . :D

Fred Berton

@fberton_twitter

hello, I would like to set up a placement strategy for grain.
In my case, I would like to be able to prohibit hosting a type of grain on a silo in cluster.
Is there a placement strategy for that? (I did not find).

spocky112

@spocky112

@fberton_twitter there isn't anything OOB but you can write custom placement logic for this https://dotnet.github.io/orleans/Documentation/grains/grain_placement.html

Harald Schult Ulriksen

@Ulriksen

@fberton_twitter https://dotnet.github.io/orleans/Documentation/clusters_and_clients/heterogeneous_silos.html ?

"On a given cluster, silos can support a different set of grain types" Seems to be what you're after

Fred Berton

@fberton_twitter

@Ulriksen Thank you. I had not seen GrainCollectionOptions in a heterogeneous page. This is exactly what I want.
builder.Configure<GrainClassOptions>(options =>
{
options.ExcludedGrainTypes.Add(typeof(IEntityGrain).ToString());
})

Fred Berton

@fberton_twitter

@ spocky112, I tried this solution. But it is very difficult because, we cannot reuse the default placement strategy (RandomPlacement, ...) as a basis for my strategy. all the Orleans strategy classes are internal, so no reuse ... and It is not possible to have access to the IPlacementRuntime interface for the same reason.

michaeltdaniels

@michaeltdaniels

I'm trying to debug an issue where stream messages sometimes don't get delivered. We're using SQS / DynamoDB / and auto scaled orleans silo instances using Fargate.

Looking at Orleans source code, I think there's a scenario where PersistentStreamPullingAgent.ReadFromQueue will cause messages to be deleted (rcvr.MessagesDeliveredAsync) before ever delivering the messages. If https://github.com/dotnet/orleans/blob/d360b539784b14a73ceb357fd6b5084d3116e9d9/src/Orleans.Runtime/Streams/PersistentStream/PersistentStreamPullingAgent.cs#L473 that line is hit, there is no guarantee that the messages added to the queue cache (on line 441) have been sent at all. A subsequent execution of PersistentStreamPullingAgent.ReadFromQueue would get those undelivered messages from queueCache.TryPurgeFromCache because SimpleQueueCache only excludes items marked as DeliveryFailure, which could only happen if there was actually an attempt to deliver the messages. The messages would then get deleted from the underlying message queue (in our case SQS) having never been delivered to the consumer.

Am I missing something, or does that sound plausible?

Jon Steinich

@jsteinich

@michaeltdaniels we experienced something very similar in testing about a year ago (dotnet/orleans#903). There was a PR earlier this year which may have fixed the issue (https://github.com/dotnet/orleans/pull/6299), but I haven't tried to reproduce since then.

David Christensen

@OracPrime

Hi there, this may be an odd question but we're looking for some advice on long running external tasks. Can some grains have a higher response timeout / turn warning threshold because we know they're going to be waiting on an external task. We've given the orleans thread back to the thread pool because we're on an await so are there any issues? It seems easier than making the external task switch context into Orleans to execute a callback after the external task is finished.

@ReubenBond do you have any thoughts on @Romanx 's question here?

MarcDHall

@MarcDHall

We have a multi-tenant situation in which we would like the ability to map tenants to storage accounts. This seems a bit tricky in Orleans. It could be done by writing a custom StorageProvider. I'm just wondering if there is (or will be) a simpler approach built in to Orleans. I mean right now you can map grain types to storage providers via the attribute but that's a constant at runtime. What I want to be able to do is have the grain determine it's provider at runtime (based on local state / deduced from it's ID).

Veikko Eeva

@veikkoeeva

@MarcDHall Just to make sure, you could use persistence from grain code without the provider. Which storage you are you using?

MarcDHall

@MarcDHall

@veikkoeeva AzureBlobs. Yeah I could do that. I would just rather stick to the Orleans patterns if possible.

Reuben Bond

@ReubenBond

@MarcDHall I think it's fine - create an abstraction around it, if you like, to load the state in your OnActivateAsync method

MarcDHall

@MarcDHall

@ReubenBond Yeah. I have a version at the moment that uses a custom 'mapped' storage provider. It works fine, tenants can be assigned to storage accounts. From the grain's perspective it's invisible, which is good, but it's a bit...so-so, architecturally. The storageProvider has to be able to deduce a storage location from the grain's id, meaning it's doing storage and mapping at once. Id rather separate those concerns so I'm thinking about a pattern where the mappedstorageprovider just redirects to other named orleans storage providers. So the tenants would be assigned to storage providers by name.

@ReubenBond also a separate question: Is it the case that streams are identified by guids only? And therefor a grainwithstringid can't have an implicit sub to a stream?

jbockle

@jbockle

unit testing question > I have a grain that uses a httpclient that is being consumed by other grains, is it possible to replace that grain's implementation in the test cluster?

Elan Hasson

@ElanHasson

Inject it via httpclientfactory

https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests

Patrick Wojcik

@pwoji

Hi all. Are there any good resources to look at regarding DI with orleans (documentation, blogs, code, etc.)?

5 replies

Mario Benitez

@zibler

Hello all. Are there any rules I should follow when asking a question in this conversation?

Mario Benitez

@zibler

Hello all. I have a use case for an application I am building using orleans and I am not 100% sure how to go about it. I was hoping to get some input from this forum.

I have several applications that I must interface to. These applications are not web based and they use UDP or TCP as their transmission mechanism. My application has several grains that must parse the received data (either UDP/TCP). I am trying to abstract away the mechanism used to send the data (UDP vs TCP). Therefore my initial idea was to have several Grain Clients in charge of dealing with TCP/UDP connections and then sending the data to the actual Grain.

My application will sit behind a load balancer and must be highly reliable and redundant. My application will in some instances push data via the client grains or in some other instances receive data and respond (poll/response).

This means that a connection, no matter who initiated it, must only be handled from a single grain client however multiple grain clients must exist in different servers so that if a server fails another grain client can take over.

My inital though was to use streams to handle the communication between the grains and the grain clients. However since only one grain client must be sending or receiving data, I really don't want to broadcast the data to all grain clients. Instead I was thinking about using the stream in a competing consummer fashion, but this is not somehting (that I know) available already in the streams provided by orleans. Things get a little bit more complilcated when using TCP as this is a connection oriented protocol. In this case, not only the data must be sent to a single client, but it also must be sent to the same grain client that currently has the active TCP connection.

I found https://stackoverflow.com/questions/55021791/orleans-custom-tcp-socket-connection this online and at the moment I am thinking of having the grain itself act as the socket server and completly eliminate the use of grain clients. Hopefully this all makes sense to you. Thank you for your time.

jbockle

@jbockle

are grain service clients only available in the silo? i.e. if my client application needs to interact with a grain service/client it would need to be wrapped inside anoter grain call?

Mario Benitez

@zibler

Grain service clients can be wherever I choose. I have full control on that. I'm was thinking to host the in the same machine where the silos run.

jbockle

@jbockle

how do I get access to the silo's IServiceProvider in a TestCluster

the TestCluster.ServiceProvider is only for the client

Han Liu

@buzzers

I have an interesting question.
I'm using OpenTracing to add tracing to orleans. At first everything was fine until I triggered a reminder.
I found that the parentid in the ReceiveReminder method inherited the span when the RegisterOrUpdateReminder method was called.
I suspect this is due to an incorrect AsyncLocal context for the scheduler.
So I tested it using RequestContext and found that The ReceiveReminder method was the same as the RequestContext in the method I used to start Reminder.
Is this a bug, or is it a known feature?

Jack Feng

@Cloud33

Hi, man，'Orleans' 3.2.1 Reminder
It seems to be invalid. This is my issues
dotnet/orleans#6632

@ReubenBond It's normal for me to go back to 3.2.0

Reuben Bond

@ReubenBond

@Cloud33 I commented on the issue to ask for more info. Hopefully we can sort it out quickly

@buzzers could you open an issue and we can work through the correct/expected behavior?

Jack Feng

@Cloud33

@ReubenBond I've added more detailed information.

Han Liu

@buzzers

@ReubenBond OK,this is #6630

MarcDHall

@MarcDHall

I'm suddenly getting Could not load type 'Orleans.Runtime.ExecutorService'. It started when I added azure queue stream storage, I think, but removing that isn't fixing it. Anyone have an idea how to resolve this?

Ashkan Saeedi Mazdeh

@ashkan-saeedi-mazdeh

Hi guys
I hope things are going well around here.
I'm back here after a very long time. We have a customer which doesn't want to use our game backend on the cloud and I'm here to see if you guys know any good storage provider on Orleans 2.x which works on .NET core as well

Where communities thrive

People

Repo info

Activity