These are chat archives for akkadotnet/akka.net

17th
Oct 2016
Arjen Smits
@Danthar
Oct 17 2016 07:07
@Arkatufus Its a tradeoff between memory usage and performance. The more you load into your actor. The more memory it consumes. If its a persistentActor that also means that it might take longer to initialise. (if you use akka.persistence there are ways to optimize by using snapshots but i digress).
That said there are ways to manage that. By using the passivate pattern to unload actors which haven't been used in a while.
Or by analysing what your 'hot' and what your 'cold' data is.
Then you can opt to more aggresively unload your 'cold' data into a database, instead of keeping them in memory inside an actor.
So there isn't one hard rule that says 'thy shall not load more then X state into your actor'
It depends on your own use-case.
Andrew Buttigieg
@andrewbuttigieg
Oct 17 2016 07:18
Hi, is anyone using the Akka.Cluster.TestKit?
Looks like it is no where near as popular as Akka.TestKit...
verilocation
@verilocation
Oct 17 2016 07:53

I have managed to get routing + cluster + remote deployment working fine. But I need to extend it so that my cluster also has access to a cluster.. When I do this i'm getting the following exception

Configuration missing for router [akka://Bifrost/remote/akka.tcp/Bifrost@192.168.0.121:56945/user/processor/c1/geocode] in 'akka.actor.deployment' section.

The GeocodeActor cluster is created within ProcessorActor so the path looks correct, but I'm not sure why its complaining as my config looks like this:

actor {
  provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
  ask-timeout = 30s

  deployment {
    /processor {
      router = smallest-mailbox-pool
      routees.paths = ["/user/processor"]
      nr-of-instances = 50
      cluster {
                enabled = on
                max-nr-of-instances-per-node = 50 # want less than this in production to ensure it spans multiple nodes
                allow-local-routees = off
                use-role = processor
            }
    }

    "/processor/*/geocode" {
      router = smallest-mailbox-pool
      nr-of-instances = 50
      cluster {
                enabled = on
                max-nr-of-instances-per-node = 50 # want less than this in production to ensure it spans multiple nodes
                allow-local-routees = off
                use-role = geocode
            }
    }
  }
}

Can anyone help? Thanks in advance

John Nicholas
@MrTortoise
Oct 17 2016 08:06
morning. So i just noticed Receive vs Async. Does this mean that Receive is now reentrant ... The wording on async seems to imply that the async method is not re entrant.
This message was deleted
verilocation
@verilocation
Oct 17 2016 08:08
I'd hazard a guess that Receive blocks the actor / current thread. ReceiveAsync won't block and will work as per standard C# async stuff which I believe is reentrant (at that particular line)
John Nicholas
@MrTortoise
Oct 17 2016 08:08
Funny, i realise now ive never had to worry about this due to stashign and become. I always designed it out
The actor will be suspended until the task returned by handler completes. is the line that caught my eye in the async comment
verilocation
@verilocation
Oct 17 2016 08:10
Thats probably because an Actor only processes one message at a time. And because, when you async a receive, itll not finish processing the current message even though its given up the thread temporarily
John Nicholas
@MrTortoise
Oct 17 2016 08:11
ahh i see what you are getting at, sorry. yes that makes sense
verilocation
@verilocation
Oct 17 2016 08:11
:)
John Nicholas
@MrTortoise
Oct 17 2016 08:11
so neither of them break the akka i'm used to ;)
verilocation
@verilocation
Oct 17 2016 08:12
I dont think so, but I'm not an expert :)
John Nicholas
@MrTortoise
Oct 17 2016 08:12
ahh but the source is nice. I'm just about 9 months behind now
John Nicholas
@MrTortoise
Oct 17 2016 08:20
@andrewbuttigieg thats probably because the non cluster testkit has been around longer combined with more people using akka core rather than clustering. Also the clustering testkit came about long after remoting so i imagine several other people figured out other ways of testing (such as starting lots of top shelf services ).
Andrew Buttigieg
@andrewbuttigieg
Oct 17 2016 08:23
Yes, I am testing my work now using the regular TestKit and I think I can do most of it quite easily.
verilocation
@verilocation
Oct 17 2016 09:05
Further to my question above. Am I right in thinking that the HOCON for a cluster should be on the node that creates it, rather than where it ends up? So in my case I have a cluster (processor) which then creates a cluster. (geocode) The config for the original cluster config (which works) is on the node (test client) that creates it. Should the config for the second cluster be on the same node (test client). Or the node that has the role upon which first cluster is deployed (processor)?
Andrey Leskov
@andreyleskov
Oct 17 2016 09:20
Hi all, I'm debugging issue with unexpected restart of ActorSystem running in Azure WebApp. I'm using Akka.Remote actor provider. Does akka.net use same AppDomain as caller of ActorSystem.Create ? Can components of akka create own AppDomains ?
Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:23
@andreyleskov actor system is just a disposable container. It doesn't know anything about app domains, so it's up to you to define, how to work with them
Andrey Leskov
@andreyleskov
Oct 17 2016 09:25
so it is running in AppDomain of ActorSystem.Create caller ?
Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:25
yep
Andrey Leskov
@andreyleskov
Oct 17 2016 09:25
ok, thank you )
Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:26
@MrTortoise actors using ReceiveAsync are not reentrant
John Nicholas
@MrTortoise
Oct 17 2016 09:27
@Horusiath Thanks, i didnt think that was the case. I was wondering if it implied Receive was reenterant though, not the async one. I realise now that neither is
Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:28
@verilocation you need to edit your question, you've used word cluster in few different meanings (basically none of them covers the cluster definition) and it's hard to catch the question
in case when you're creating clustered router, the config location that matters here is where you're calling ActorOf for that router.
@MrTortoise if you're using Receive + PipeTo pattern, then that actor is reentrant
verilocation
@verilocation
Oct 17 2016 09:33

lol ok sorry... let me rephrase. If you look up a little youll see my initial question.

I have a test client that contains all code. It contains the HOCON mentioned above. At first it creates a ProcessActor which is defines as a clustered pool. This is remotely deployed onto a node which defines its roles as [process, geocode]. The Process actor (running on the node) then attempts to create its own clustered pool for geocodes, which just happens to be deployed to itself at the moment. Its this bit that doesnt work properly. The config for the geocode is on TestClient (I was attempting to copy how webcrawler defined this)

Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:35
@verilocation not sure, if you're not overcomplicating things, but in this case you need to place geocode router config on every node, where processor actor may call it
verilocation
@verilocation
Oct 17 2016 09:40

Right.. ok that works, thanks... Perhaps I am over complicating things? What would be the recommended way of doing things.

If I create a bunch of ProcessActors to process inbound messages. Would you expect them to create one actor each to work with. Or (in my case at the moment) each ProcessActor defines a pooled cluster of [geocode] to use which, I agree, is more complicated

Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:42
I don't know what are your data relationships (and I'm not a big fan of routers), but maybe it would be better to have processes and geocodes as two different pools with reference to each other
verilocation
@verilocation
Oct 17 2016 09:45

That sounds reasonable, how can I do that? Currently within my processor I create the geocode actor pool like so

this.GeocodeActor = ReceiveActor.Context.ActorOf(Props.Create<GeocodeActor>().WithRouter(FromConfig.Instance), "geocode");

Which (thanks to your help) I now know is taken from the node that the Processor is running on, not the test client. Would I be able to define them both in the test client but just let them know about each other?

(P.s. any reason why youre not a fan of routers?)
Bartosz Sypytkowski
@Horusiath
Oct 17 2016 09:57

I don't like routers, 'cause they introduce too much "magic". If they were fully verified at compile-time, this wouldn't be much of an issue, but as everything is driven by HOCON config at runtime, it's easy to get hard to debug runtime errors.

Regarding ways to communicate:

  1. You can add some roles to cluster nodes to recognize the nodes, where process and geocode actors lives. Then on the each side, you can subscribe to something called cluster events - these are evens triggered when new node becomes visible to the current actor. With that you can filter out proces/geocode nodes and use actor selection to get references to them.
  2. Second option is to use distributed pub/sub - it's a publish subscribe mechanism that works across cluster nodes. So geocode actors simply subscribe to some topic, while process actors will publish to it.
verilocation
@verilocation
Oct 17 2016 10:07
Thank you I shall look into it :)
verilocation
@verilocation
Oct 17 2016 10:51
Quick Q... we are a small team (4 developers) but we are starting to run into App.Config issues relating to public IP addresses. Is there any recommended method for dealing with different IPs in different App.Configs for different developers? Because when we check in quite often developers will check in the App.COnfig which will mess about with other developers setups as itll overwrite the IP addresses
Peter Bergman
@peter-bannerflow
Oct 17 2016 11:14
Maybe IPs in the config could be avoided? What is the reason for having them there?
verilocation
@verilocation
Oct 17 2016 11:15
Currently, on our dev machines, we use our public IP so that we can [a] develop independently and then [b] easily tell our lighthouses to talk to each other to check everything works in a distributed manner
Peter Bergman
@peter-bannerflow
Oct 17 2016 11:22
Not sure about the setup but when I develop locally couldn't you connect to a local lighthouse as well? And just connect all nodes on loop back as hostname?
I meant when you develop locally, not I :)
Peter Bergman
@peter-bannerflow
Oct 17 2016 11:31
But if you really need the IPs in there to do networked things with other developers then maybe the IP could be injected to the hocon at runtime instead.
Gregorius Soedharmo
@Arkatufus
Oct 17 2016 11:58
i thought only lighthouses need predefined ip addresses, and the other nodes in the cluster find each other out through gossip?
verilocation
@verilocation
Oct 17 2016 12:12
I find that if I dont specify the public hostname on all my nodes, then even though the lighthouses talk to each other and every node discovers each other, I tend to get messages like (and im paraphrasing) "Tried to send message to [public ip] but its being discard because the node is only listening on [local ip]"
But yes @Arkatufus I only make my seed / lighthouse nodes the ones that my other nodes connect to
Gregorius Soedharmo
@Arkatufus
Oct 17 2016 12:15
@verilocation oh, so its a problem with nodes behind router NAT?
verilocation
@verilocation
Oct 17 2016 12:15
Honestly not sure, it seems that the lighthouses think the messages arent bound for the right place so discards them
Francis Paulin
@paulinfrancis
Oct 17 2016 12:35
I just implemented an extension method on IConfiguration (ASP .NET Core), so that you can have your hocon as json in appsettings.json:
public static class ConfigExtensions
{
    public static Config ExtractAkkaConfig(this IConfiguration section, dynamic configSection = null)
    {
        if (configSection == null)
        {
            configSection = new ExpandoObject();
        }

        var expandoDict = configSection as IDictionary<string, object>;

        var configurationSections = section.GetChildren().ToList();
        foreach (var configurationSection in configurationSections)
        {
            if (configurationSection.Value != null)
            {
                expandoDict.Add(configurationSection.Key, configurationSection.Value);
            }
            else
            {
                var exp = new ExpandoObject();
                expandoDict.Add(configurationSection.Key, exp);
                ExtractAkkaConfig(configurationSection, expandoDict[configurationSection.Key]);
            }
        }

        var configJson = "{ akka: " + JsonConvert.SerializeObject(configSection) + " }";

        return ConfigurationFactory.ParseString(configJson);
    }
}
verilocation
@verilocation
Oct 17 2016 12:35
Very nice
Francis Paulin
@paulinfrancis
Oct 17 2016 12:36
is there an "extensions" repo I could submit a PR to? I am probably neither the first nor the last person to want to use Akka in Core :)
Ash R
@AshRolls
Oct 17 2016 15:15
I'm sorry if this is the wrong place to ask, but I have an actor system and I'm not getting the throughput I would expect? 1 top level actor, with 1000 child actors. Each child actor has another 50 child actors. (51001 actors in total) The top two levels listen for all of their respective child actors to send back a message. All messages sent with tells. To send a message 100 times from top to bottom (and then back) takes around 10 seconds. The code is here.... https://github.com/AshRolls/GalaxyGen/blob/akka_speed_experiment_2/GalaxyGenActorPerformanceTester/DefaultTellCounter/DefaultTellCounter.cs
I've tried playing with dispatcher throughput
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:17
@AshRolls tough to tell what the "throughput" should be without hardware
Ash R
@AshRolls
Oct 17 2016 15:17
quad core i5-3570k
John Nicholas
@MrTortoise
Oct 17 2016 15:17
whats the recommended goto Receive handler these days?
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:18
but just grokking from where the wind blows here
10 seconds to do that sounds reasonable, and I'll explain right now
the biggest cost to processing a message, usually other than the work the actor actually has to perform when it receives a message, is the context switch on the dispatcher
Ash R
@AshRolls
Oct 17 2016 15:20
Yes, that was my assumption
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:20
in your case there isn't that much work since it looks like each child is just sending a basic ACK back
Ash R
@AshRolls
Oct 17 2016 15:20
yup this is just the test case. I thought the context switching was taking the time, hence the fiddling with custom-dispatcher throughput
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:21
the dispatcher throughput can't and won't affect the context switching overhead in this scenario
what the dispatcher throughput will do is allow one actor to hang onto the dispatcher for longer before it yields to another
but if each actor only has to process a single message at the lowest level of the hierarchy
Ash R
@AshRolls
Oct 17 2016 15:22
so back of the hand 1 mill messages per sec on my hardware is around the correct value?
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:22
the other thing about dispatcher thoughput
it only increases the throughput of a given actor if you adjust that setting exclusively
if everyone's throughput setting goes up, no one's does
(insert exceptional edge cases here where the system isn't that busy, etc)
but in a busy system, that's true
so typically where you see a lot of benefit in increasing the dispatcher throughput is when you use it on a small number of actors who receive a disproportionately large number of messages relative to everyone else
Ash R
@AshRolls
Oct 17 2016 15:24
I'll try and optimise the architecture to reduce context switching then
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:24
on our CI system we run on a crappy Windows Azure VM
D2 V2
has two cores
a single actor can do about 4.3-4.5 million messages per second
but that's because its dispatcher is constantly running
Ash R
@AshRolls
Oct 17 2016 15:25
I saw the pingpong example mentioned in f#
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:25
yeah, the ping pong example is a better benchmark for looking at context switching costs
they start to show up once the number of actors running exceeds the number of cores by some scalar value
in a typical Akka.NET system in production
you have a large number of actors, but at any given instant only a proportion of them have messages to process
the majority are idle, if you took a snapshot at any given millisecond
Ash R
@AshRolls
Oct 17 2016 15:27
thanks for your help, I have some ideas now
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:27
excellent, glad to hear it
Ash R
@AshRolls
Oct 17 2016 15:27
reduce messages by batching
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:27
please feel free to ask questions here or on StackOverflow
Ash R
@AshRolls
Oct 17 2016 15:28
ty, very helpful. loving akka.net so thanks for the devs work on it
Aaron Stannard
@Aaronontheweb
Oct 17 2016 15:29
thanks!
We have some cool stuff coming out again soon too
verilocation
@verilocation
Oct 17 2016 15:30
I'd second that. Also it's nice to have such good support through this!
Ash R
@AshRolls
Oct 17 2016 15:46
I just had nice speed boost by not creating a new object each time I need to create a message to send / return
Juan José De Arana
@juanjoarana
Oct 17 2016 15:47
Hello, a colleague complained that it's not that easy to design an Actor model, where you can make any logical unit an actor. Specially when local message passing requires serialization, which I guess means a performance penalty. Is it possible to configure AKKA.NET to avoid message serialization when passing messages between actors in a local system? E.g.: passing messages objects by reference (ensuring they are immutable)... or doing a deep copy/clone of the message object? Thanks!
verilocation
@verilocation
Oct 17 2016 15:54

The other day I asked a question regarding Ask... Someone kindly showed me a related link (http://bartoszsypytkowski.com/dont-ask-tell-2/). I have since been trying to position myself whereby I no longer have to use Ask but have come up against an issue.

Previously:

  • My ProcessActor Asked the GeocodeActor
  • The GeocodeActor Asked the GeocodeCacheActor for a cached geocode response, if it has one, Tell back to the ProcessActor
  • Failing that it would Ask the ReverseGeocodeActor to reverse geocode against Google, and then Tell to the GeocodeActor who would in turn Tell to the ProcessActor
  • At this point the GeocodeActor would Tell the GeocodeCache to cache the result

Since then I now do the following

  • My ProcessActor Tells the GeocodeActor to geocode
  • The GeocodeActor has a local cache, if its hit itll Tell the sender the geocode (the sender, ProcessActor, has a receive event for this, and stores a list of pending models)
  • If one doesn't exist it will Forward the request to ReverseGeocodeActor to reverse geocode. This would ideally then Tell the response back to the original sender, the ProcessActor.

Unfortunately I can't figure out how to cache the response without an Ask as the ReverseGeocodeActor will see that the sender is the original ProcessActor...

Ash R
@AshRolls
Oct 17 2016 15:58
instead of forwarding why don't you get the GeocodeCacheActor to Tell the ReverseGeocodeActor, and send it the IActorRef of the process actor as a prop. The ReverseGeocodeActor can then send the response to the ProcessActor, as well as a Tell back to the GeocodeCacheActor with the data to cache?
verilocation
@verilocation
Oct 17 2016 16:11

Ash dont suppose you have a code example? Currently I do this in the GeocodeActor when I ask the ReverseGeocode

this.ReverseGeocodeActor.Forward(p);

Which then Tells back to the ProcessActor like so

this.Sender.Tell(geocodedAddress);

In this case how would I tell the GeocodeActor to cache the result

Ash R
@AshRolls
Oct 17 2016 16:14
In the geocode actor, take the contents of the message p, and put it in a new message type which contains those contents, as well as an IActorRef to Sender (which at this moment will be the ProcessActor). Then Tell that message to the ReverseGeoCode actor.
verilocation
@verilocation
Oct 17 2016 16:14
ahhhh
I understand you now... Does it not seem.. wrong.. to store the sender in a POCO rather than use akka somehow itself.
Ash R
@AshRolls
Oct 17 2016 16:15
In the ReverseGeoCode actor, you can then Tell Sender (GeocodeCacheActor) with the data to cache, and you also have the IActorRef so you can send ANOTHER message to tell the ProcessActor
as for structure... can't say sorry, I have been sending IActorRef around in my messages without problem, it seems pretty lightweight
if it's a set structure you can send it just once in the constuctor of the agent
verilocation
@verilocation
Oct 17 2016 16:17
Oh now that sounds better
Ok that works a charm, I like that, thank you
Ash R
@AshRolls
Oct 17 2016 16:21
np
Marc Piechura
@marcpiechura
Oct 17 2016 16:22
@juanjoarana Akka.net only serialize messages if they are send over the wire and we use the object ref and don't do deep cloning because that would hit the performance, so we have by default the requirement for immutable messages
Andrew Young
@ayoung
Oct 17 2016 16:55
@Horusiath issue created #2340
John Nicholas
@MrTortoise
Oct 17 2016 18:29
@AshRolls re not creatign messages: so you used some kind of static instance? or do you mean you had a pool of messages to use?
himekami
@himekami
Oct 17 2016 21:49
Hi, anyone knows how to create cluster singleton using FSharp API ? What should I pass to SpawnOption list (I'm guessing i have to use SpawnOpt) ?