These are chat archives for akkadotnet/akka.net

14th
Aug 2018
Bartosz Sypytkowski
@Horusiath
Aug 14 2018 06:34
@boekabart are you using default sql server journal?
@AndreSteenbergen it's always good to open source such things. But keep in mind that this doesn't mean, that akka team will have enough time to maintain it for you ;P
Lutando Ngqakaza
@Lutando
Aug 14 2018 06:52

What are the pitfalls of using inmem cluster sharding for the event journals and snapshot journals in a multinode-clustered scenario.

docs read
persistence (default) depends on Akka.Persistence module. In order to use it, you'll need to specify an event journal accessible by all of the participating nodes. An information about the particular shard placement is stored in a persistent cluster singleton actor known as coordinator. In order to guarantee consistent state between different incarnations, coordinator stores its own state using Akka.Persistence event journals.

so that means there are possible reincarnations of singletons in your cluster (ie singletons wont work as advertised)
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 07:21
@Horusiath I know, I am testing it locally at the moment. There are some issues I am having, but after I resolve those I'll open source it.
I'm just asking because I know when I open source stuff like that, I will get questions and I will get issues, I only want to open source stuff when I know therre is some need for it somewhere. And I don't get into trouble business wise off course.
Bartosz Sypytkowski
@Horusiath
Aug 14 2018 07:51
@Lutando there can be only one cluster singleton at any given time, however you shouldn't use in-memory journal anywhere outside of tests. If your cluster singleton will die and get reincarnated, it will lost all of the information about where particular shards live. You can mitigate that by using ddata mode for akka cluster sharding (which also uses in-memory store, but it's replicated and synchronized between nodes). ddata mode however doesn't allow you to use remember entities sharding option (yet).
@AndreSteenbergen you can always warn that this thing is just open sourced, but not promised to be maintained ;P
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 07:59
Good idea, thx never thought about that
I'll close the pull request I made for streams, as I think you are correct, events are not really what is desired
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 08:09
Events like these can cause race conditions as well, all source events could not have been processed, while the event is already given.
Bart de Boer
@boekabart
Aug 14 2018 08:27
@Horusiath I'm not aware of picking another one, yes. Just the journal from the nuget package. Same behaviour on 1.3.5 and 1.3.8 btw.
Lutando Ngqakaza
@Lutando
Aug 14 2018 09:05
awesome thanks :) just wanted to confirm that this was the case.
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 11:21
I would like to stop a source, which doesn't end by itself. I have found something for Java akka here: https://stackoverflow.com/questions/31378978/how-to-properly-stop-akka-streams-from-the-outside but I am a bit clueless how that would work in Akka.Net, anyone an idea?
Never mind, KillSwitch to the rescue I guess ....
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 12:39
Is it possible to create an untypedactor in my dll, but when someone wants to use the actor in there own project to add Persistence at that moment?
Or would I need to create 2 separate actors, one which is persistent, the other just an untypedactor?
Bart de Boer
@boekabart
Aug 14 2018 12:51
@AndreSteenbergen The latter - but what you could do, is create a class that someone can 'pull into' their Actor and (easily?) hook up to the Receive (and possibly other) method(s). Composition over inheritance...
AndreSteenbergen
@AndreSteenbergen
Aug 14 2018 12:52
I like that thought Bart
Basically just some class that can handle all events and commands
So the actor would be a wrapper of the class I provide
Shukhrat Nekbaev
@snekbaev
Aug 14 2018 15:00
hm, if I run the test alone - it passes, however, if I run two different tests for the same actor, then it doesn't, even though tests supposed to be executed sequentially. Running via VS + Resharper nunit3 test runner. This passes when only single test is executed:
instanceActor.Tell( startProcessing );
IgnoreMessages<Messages.InstanceActor.NotifyProcessingFinished>();
ExpectTerminated( membershipBindingSyncOrchestrationActor );
however, same logic when two tests are selected to run fails with "Expected a message of type Akka.Actor.Terminated, but received {REDACTED.Messages+InstanceActor+NotifyProcessingFinished}"
yet, if I do:
IgnoreMessages<Messages.InstanceActor.NotifyProcessingFinished>();
instanceActor.Tell( startProcessing );
ExpectTerminated( membershipBindingSyncOrchestrationActor );
then both ways of running are ok
Shukhrat Nekbaev
@snekbaev
Aug 14 2018 15:05
instead of IgnoreMessages tried ExpectMsg<Messages.InstanceActor.NotifyProcessingFinished>() and it seems to also solve the matter. Is there something special about IgnoreMessages or did I just PEBKACed somewhere? :)
Vagif Abilov
@object
Aug 14 2018 16:56
I spent last days figuring out why we exerience perstistent actors recovery failures under high CPU load, I thought this was due to the database slowness, but apparently the slowness comes from recovery itself, database calls complete quickly. We have configured ask-timeout to be 60 seconds, and recovery-event-timeout was at its default values (30 seconds). Then if the number of simultaneously recovering persistent actors exceeded 50-60, recovery often failed with either Ask timeout or errors like "Didn't get snapshot within 30s". And the SQL calls duration rarely exceeded few seconds (mostly taking milliseconds like it should be for actors with no or little history in event journal).
After confirming that the error is not caused by database layer (we are using SQL Server), I restricted number of concurrent recoveries, and when it doesn't exceed 20-30 there are no more timeouts.
I wonder if anyone has seen similar behavior, and if you are using actor persistence, is your system capable of simultaneously handling large number of persistent actors recovery?
PinkyBrain
@PinkyBrain_gitlab
Aug 14 2018 18:53
what is the recommended practice for a simple non-persisted, not clustered, actor system within asp net core? should the actor system be registered as a singleton? as a hosted service? should it live completely outside the asp app?
Aaron Stannard
@Aaronontheweb
Aug 14 2018 19:59
@object yeah, seen some of that myself using some other Akka.Persistence implementations
non-SQL Server
so it must be something within Akka.Persistence itself
Vagif Abilov
@object
Aug 14 2018 20:41
Strange that it starts timing out and failing at a relatively low level of concurrency. I guess the lesson is to use cluster so aggregate roots that use persistence are spread around multiple nodes.
Bart de Boer
@boekabart
Aug 14 2018 20:44
I was reading the Persistence docs yesterday and you can reduce the number of concurrent recoveries, maybe that helps?
@Aaronontheweb or @Horusiath - update on the Persist-fail issue with SqlServer:
  • Connection errors lead to PersistRejected (bug I think) and not stopping actor (good I think)
  • When there is a persist conflict (start same non-cluster app twice with same config), PersistFailure and stopping actor (2x good I think)
Vagif Abilov
@object
Aug 14 2018 21:00
For the time being I reduced the number of concurrent recoveries by reducing degree of parallelism in the part of system that uses persistent actors. Are there other options?
Bartosz Sypytkowski
@Horusiath
Aug 14 2018 21:02
@object akka.persistence.max-concurrent-recoveries (50 by default)
Vagif Abilov
@object
Aug 14 2018 21:37
@Horusiath Ah this one. I didn't change the default and there were lots of timeouts. Now I changed reduced the throughput of RabbitMQ channel that sends messages to the system. But this setting is another way of restricting it.