These are chat archives for akkadotnet/akka.net

6th
Dec 2016
Maxim Cherednik
@maxcherednik
Dec 06 2016 08:49
Hi guys, is this chat an appropriate place for akka(actor model) design questions or I'd better go somewhere else ?
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:20
@maxcherednik sure, you can ask here
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:33

Still investigating on this:

Hi! I got a child actor failing, the parent provides a supervisor strategy for that exception of Restart. And then failing actor instance executes PreRestart and PostStop - but no new instance is created, no call of the constructor, PostRestart or PreStart. Has somebody a hint how the restart might fail?

Basically it looks like the actor system begin to do the restart of the actor because PreRestart and PostStop are called - but the new instance is not created and therefore not swapped in
Maxim Cherednik
@maxcherednik
Dec 06 2016 09:38
@Horusiath Thx. This weekend I was trying to go a bit further rather than Hello world example. I am kind of confused about Actor re-usability and composability. Let's I have TradeHolderActor which implements some simple logic, but after all it contains a dictionary with trades. So imagine, now I have several reports on top of this Actor. One needs to count trades, another one needs to sum up pnl for the specific instruments only and so on. So I was thinking to have an Actor type per report - cause I don't want to include different kind of reports into the original actor - it will get dirty very fast. Having all of this, my original Actor supposed to have some kind messaging protocol so that to share it's state with the reporting actors. This way if those reports are on demand, I need to transfer all the state to the reporting actor - lets say 10 mln rows(wow! every single time to calculate, i will need to transfer all the state). No imagine, I need to calculate this every 3 seconds. Obviously I transfer the state all the time. So this pushes me to make somehow a pub sub in the original actor, so that to spread the state to the reporting actor. But that means that I will double the memory usage, since I will have a copy of the data on the reporting actor. Am I missing something?
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:38
The supervisor strategy looks like this:
        protected override SupervisorStrategy SupervisorStrategy()
        {
            return new OneForOneStrategy(
                maxNrOfRetries: 3,
                withinTimeRange: TimeSpan.FromMinutes(5),
                decider: Decider.From(x =>
                {
                    if (x is ActorInitializationException) return Directive.Stop;
                    if (x is ActorKilledException) return Directive.Stop;
                    return Directive.Restart;
                }));
        }
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:39
@ZoolWay On which exception are you restarting (custom one?). You could also write log event when your actor starts
to make sure it's restarting correctly
if maybe your actor is failing in when constructor get's called upon restart?
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:41
@Horusiath I got logs in the constructor, PreStart and PreRestart but none gets called after the failure. I am throwing a simple System.Exception which should trigger Directive.Restart - the decider is evaluated.
There is a log statement at the first line of the constructor which is not triggered
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:43
@ZoolWay do you have this code somewhere? Any repro?
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:43
Unfortunately not. If I cannot find something I will try a simplified example on GitHub
The constructor has parameters... the actor system has saved them from the first call, doesn't it?
I am not using any kind of DI
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:44
constructor parameters are resolved from Props
when you're creating actor using props, they are stored internally and used later for restarts
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:46
Yeah, I got var propsChild = Props.Create(() => new FailingActor(Self, this.configdata, this.masterActor)); this.childWorker = Context.ActorOf(propsChild, "worker");
So the props references something in the parent actor obviously. Can that be the problem?
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:49
it looks a little like state sharing, but if it's immutable, it shouldn't be an issue
@maxcherednik you can have one actor (your TradeHolder) having a write state, and report actors with a read state. They are subscribing to the trade holder. When TradeHolder state changes, publish changeset of it to every report actor and make them update their read state based on the provided data. This way actor are updating only state changes not the whole state, and are updated live, as the changes happen, not when requested.
Ricky Blankenaufulland
@ZoolWay
Dec 06 2016 09:50
the config data does not change, the rest is giving actorrefs to have the child easily send them messages without using ActorSelection
ok, as it looks like I am not doing something obviously evil, I will take a look at a minified example. thanks so far @Horusiath
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 09:52
np, let me know if you get any progress on that
Maxim Cherednik
@maxcherednik
Dec 06 2016 09:55
@Horusiath So we are on the same page then.... and I guess I can't do anything about the fact that there's a big memory footprint.... Right?
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 10:00
@maxcherednik if you need that data often, there's not much you can do about it ;)
but maybe you don't need to store every single trade - can I assume you're familiar with eventsourcing?
Maxim Cherednik
@maxcherednik
Dec 06 2016 10:02
@Horusiath yeah, I thought so :) either lots of cpu and time or prepared state so that to react faster, but the price is memory.
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 10:03
because you've persist your data using eventsourcing semantics and keep in memory only that state that is actually in use (not every trade i.e. but only multiple aggregated results)
Maxim Cherednik
@maxcherednik
Dec 06 2016 10:03
sure
Maxim Cherednik
@maxcherednik
Dec 06 2016 10:11
@Horusiath may I ask one more question... it's kinda related.... I was looking into akka streams. Is my understanding correct that it's one time passing logic? What I mean is this. Take my previous example. I have a chain which is subscribed to my TradeHolder. But the semantics of the stream not just push, push, push, but more like add, update, delete. So trades could be deleted. So imagine I have a Filter link, which was bypassing some types of trades. So then I receive update or delete of the trade which was previously sent through the Filter. So this filter supposed to keep the state of the previously sent items and if something changed it needs to send the corresponding add, update or delete further through the chain. I guess I cant have this with the current implementation of the Akka streams, right?
Do I need to implement the whole logic manually?
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 10:13
add/update/delete can be just different types of messages to be sent.
you can send messages from within the akka streams to other actors, but I don't think it's a good idea
Maxim Cherednik
@maxcherednik
Dec 06 2016 10:15
it seems to me that it's not really possible out of the box
I need to implement all of this manually
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 10:16
imagine if someone it using your filter which replays given data, as reaction it may send some message to another actor. But if something bad happens, this filter will need to replay that stream of data again. Now it may send the same reaction message second time even when it was not necessary
Maxim Cherednik
@maxcherednik
Dec 06 2016 10:19
yeah, I see... but I guess it could be organized in the parent/child way.... if link in the middle restarts, then all the links downwards restart as well
anyway... my original question is about the semantics of the stream... as far as I understood the Out channel should be one message type only, is this correct?
similar to reactive extensions of the .net iqueryable<T>
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 10:22
yes, but behind that type there may be a multitude of implementations
Alex Michel
@amichel
Dec 06 2016 11:14
when using cluster sharding for millions of entities, is there some best practice to set number of shards? Another question is whether it makes sense to shard consistent hash routers than handle scale out to entities rather than entities themselves?
Aaron Stannard
@Aaronontheweb
Dec 06 2016 16:02
@amichel the MessageExtractor implementation will let you set the maximum number of shards
and I believe it has a consistent hashing base class you can use, or am I wrong about that @Horusiath ?
Alex Michel
@amichel
Dec 06 2016 16:08
@Aaronontheweb I see. What I asked is whether it is common to set small number of shards or I can set shard id = entity id and have millions of shards?
@Aaronontheweb my second question was whether Shard that manages Dictionary of ActorRef to route relevant messages to entities is as efficient as ConsistentHashPool of actors (assuming I have only one shard) for same number of entities and message rate
Aaron Stannard
@Aaronontheweb
Dec 06 2016 16:16
ah I see
you're asking about what's the most efficient way to route messages to the shard entity actor's children?
make the shard entity actor a custom one with a built-in dictionary
or have it use a pool?
second question: a shard represents a collection of entities
each of those entity actors will be named after the entity id produced by the MessageExtractor
which is mapped from the messages you send to it
similar to extracting the hash key for consistent hash routing
IMHO, if you know that you're going to have a lot of entities
better to have a larger maximum number of shards
you end up with more shards to manage
but the impact of rebalancing is much lower
not affecting as many entities at once
if you only have 10 shards
each one can end up owning a huge number of children if you have a massive number of entity actors active at any given time
I don't have direct experience with performance-tuning that under load yet, but I'd imagine having a shard count that's higher (100 or 1000) would probably be more stable in the event of a rebalance of a very large entity set
Aaron Stannard
@Aaronontheweb
Dec 06 2016 16:21
could you double check me on that previous statement @Horusiath ? am I speaking nonsense :p
Alex Michel
@amichel
Dec 06 2016 16:26
@Aaronontheweb if I have two nodes, no matter how many shards I create, I will need to rebalance on average 50% of entities.
if one of nodes fail
@Aaronontheweb on the other hand if cluster is stable and under pressure I need to create more entities, dictionary lookup becomes slow inside shard at some point if you have too many objects in it (due to chaining in hashtable)
Alex Michel
@amichel
Dec 06 2016 16:31
@Aaronontheweb 100 shards sounds reasonable for 1M entities I guess, but would be nice to hear from someone who tested it in real app
Alex Michel
@amichel
Dec 06 2016 16:40
@Aaronontheweb I assume that main disadvantage of cluster sharding vs clustered consistent hash router is persistence/journaling overhead?
Aaron Stannard
@Aaronontheweb
Dec 06 2016 16:55
@amichel yeah there's some overhead in a couple of places
but it's more intelligent than a CHR
knows to buffer messages for shards that are being moved
which helps solve a really nasty class of consistency problem
which you'd have to figure out how to do yourself using a CHR
Alex Michel
@amichel
Dec 06 2016 17:13
@Aaronontheweb thanks a lot. Will give a try in different workloads
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 17:50
@amichel shard per entity is a bad idea - too much overhead. Using hashing message extractor you must specify some threshold (rule of thumb says it could be 10 * max nr. of nodes in your cluster). But if you have some natural identifier of some sort (like office-id. region-id etc.) you could use it as shard id.
and the speed of creating nodes is probably highly dependent on the type of the persistent storage you'll use.
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 18:03
There are 2 changes we probably will improve on that field quite soon: first one is new faster journal implementation for SQL-backed storages. Second is a different approach to updating internal cluster sharding state using CRDTs instead of eventsourcing.
Aaron Stannard
@Aaronontheweb
Dec 06 2016 18:48
ah, good to know about the rule of thumb
did not know that
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 18:49
found that on official akka jvm docs :P
Rich Cox
@conejo
Dec 06 2016 18:53
I'm using Akka.IO to implement a simple TCP listener / handler. It's working well so far, but I need to enable TLS. Any tips or ideas how I can do that?
Alex Michel
@amichel
Dec 06 2016 18:55
CRDT should definitely reduce bottleneck and single point of failure of shared storage. Is it going
Is it going to be internal implementation or over Riak or something like that?
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 18:58
@amichel it's a custom module we've ported from the jvm - it's in-memory state-based and contains implementation of replication protocol, prunning and some CRDTS.
but I know that this month people from Lightbend are working on hardening their implementation - they are modifying it into delta-based CRDTs with persistence suport (as I've seen they have chosen LMDB as a default persistence layer)
Alex Michel
@amichel
Dec 06 2016 19:21
This could allow to implement a memory grid. And possibly geo cluster
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 20:51
@amichel akka cluster membership protocol already works using form of CRDTs. It's possible to configure cluster to run on multiple DCs already
additionally once #2330 will get merged we could loose member visibility constraints. It would allow us to greatly increase cluster size in situations of high node churn
Bartosz Sypytkowski
@Horusiath
Dec 06 2016 21:06
@conejo when we fix remaining parts of akka streams tcp flow, you'll get better semantic to work over that. Unfortunatelly at this point I don't know if you'll find a ready solution. I know that spray has some actor which creates a ssl wrapper over raw akka.io, but it's possible we could do it easier, as JVM and .NET way of dealing with secure connections differ quite significantly.
Rich Cox
@conejo
Dec 06 2016 21:13
Thanks @Horusiath. I'll poke around and look at spray and see if that could be adapted. I was hoping for something easier :-). We might also look at a fallback plan and have NGINX or something else to the TCP TLS termination for us.
Alex Achinfiev
@aachinfiev
Dec 06 2016 23:22
Question about TestKit. I am trying to send a number of messages to a persistent actor which fires an event on the system event stream after an entity is persistent. Is there a way for me to ensure that if I send 10 requests, that all 10 will fire within a given timeout? I tried using EventListener.ExpectMsgAllOf(timeout, expected[]) but that requires me to know the results ahead of time. There is a method private IReadOnlyCollection<T> InternalExpectMsgAllOf<T>(TimeSpan max, IReadOnlyCollection<T> messages, Func<T, T, bool> areEqual = null, bool shouldLog=false) in TestKitBase which allows to pass a Func for the equality but that's not exposed outside and no other method uses it. Which approach is used for the cases like this? Thanks.