These are chat archives for akkadotnet/akka.net

27th
Jul 2016
Bart de Boer
@boekabart
Jul 27 2016 04:48
Modifying state in continuation (after await) is indeed bad, typically you want to just 'Tell' someone or yourself the result.
qwoz
@qwoz
Jul 27 2016 05:08
And if you're not modifying the current actor's state, no issues with going async all the way?
eg: gather info from other actors, then Tell the result to another actor
Ricky Blankenaufulland
@ZoolWay
Jul 27 2016 07:13
Let's say I got a user-coordinator actors which should have a child user-XXX for every user logged in. When the coordinator receives a messages it will forward it to the child. But if there is no child, it has to ask another actor for some more user-data before it can create the child. How can this pattern be improved without Ask / await? I am now doing it like @qwoz said with some ContinueWith/PipeTo which means if I want to create the child I send a message for requesting the information and then send myself a message to continue. So the process of forward to child actor is interrupted, the message handler ends in that case and another message handler will do the second part - which is forwarding the original message which will already be done if the child already existed in the first place.
Toshko Andreev
@Ravenheart
Jul 27 2016 07:15
i'd make the child and have in a state of creation
have it fire a message for the extra data
when the data message comes switch to the proccesing state and begin normal work
Ricky Blankenaufulland
@ZoolWay
Jul 27 2016 07:17
What about the message the coordinator would immediately forward to that child?
Marc Piechura
@marcpiechura
Jul 27 2016 07:17
You could also send the message to the data actor from the coordinator and set the created actor as sender, this way the child doesn't need a reference to the data actor
Stash all messages until the message from the data actor arrives
Ricky Blankenaufulland
@ZoolWay
Jul 27 2016 07:19
Hmm, this sounds very interesting for this kind of scenario. I'll give it a try, thanks!
For the message-based approach I still need to adopt my thinking from time to time
qwoz
@qwoz
Jul 27 2016 07:55
yep, I'm also using the stash approach. Actually I'm doing it one of two ways. Either load the data the actor needs in its constructor before calling the method that implements all the Receive handlers. So effectively all the messages block until that completes, then those messages process and the data is there. Or in the constructor do something like Become(Configure) and fire off a few Tell messages to get the info it needs. Inside the Configure method, you have things like Receive<ConfigSettings>(...); Receive<OtherStuff>(...); Receive<YetMoreThings>(...); and lastly ReceiveAny(msg => { Stash.Stash(); });
When you've received all the config items you need, do a Become(HandleMessages); Stash.UnstashAll(); and all those messages that were squirreled away are resent to itself.
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 08:17
Hello does Akka.Net IO Api provide stuffs like codecs and handlers or one has to implement them him/herself?
Ricky Blankenaufulland
@ZoolWay
Jul 27 2016 08:20
@qwoz @Ravenheart @Silv3rcircl3 Okay, that feels like a much better approach. Also with ReceiveAny this looks a lot more easier for stashing :D The actor will need to implement IStash, won't it?
qwoz
@qwoz
Jul 27 2016 08:22
yes... public class YourActor : ReceiveActor, IWithUnboundedStash
with member public IStash Stash { get; set; }
Ricky Blankenaufulland
@ZoolWay
Jul 27 2016 08:24
Thanks!
Peter Bergman
@peter-bannerflow
Jul 27 2016 12:24
If I want to investigate the size of the messages I am sending to remote nodes, what would be an accureate approach? Just serialize the message myself one time before using it in the Tell statement? Or is there some other way to go about it?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 17:53
@peter-bannerflow you can turn on remote metrics inside HOCON
that will start logging warnings for messages larger than the byte size you specify
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:08
for folks who are all taking a fresh look at Akka.Cluster as of 1.1, I wrote this yesterday to provide some explanation for how state distribution in a cluster typically works using the consistent hashing router: https://petabridge.com/blog/akkacluster-state-distribution/
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:12
Hello does Akka.Net IO Api provide stuffs like codecs and handlers or one has to implement them him/herself?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:13
nope
you have to write those yourself
Akka.IO gives you IO capabilities and that's it
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:13
Ok
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:13
message framing, codecs, and so forth have to be implemented yourself
you don't really need a handler per-say
because the messages are all being handled by an actor
so the 1 message at a time guarantee makes things a bit simpler
but you might still have to deal with issues like partial reads
since the IO system is dealing in raw bytes
not messages
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:14
@Aaronontheweb That is cool.
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:14
yep
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:15
Will be a good advice to implement Tcp protocol related stuffs using DotNetty and the business logic using Akka.Net?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:15
http://blog.stephencleary.com/2009/04/message-framing.html - good explanation from Stephen Cleary on message framing
yep, I would
assuming that
your wire protocol is sufficiently complex
i.e. you're using something like MQTT
if you're doing something basic
like just passing simple frames over TCP
then Akka.IO will do that just fine
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:16
ok
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:19
actually I like our explanation of message framing better https://petabridge.com/blog/large-messages-and-sockets-in-akkadotnet/
I included some diagrams to show how those strategies work
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:24
What about I am implementing a FreeSwitch library? The reason I am thinking that is because I have implemented the library using DotNetty. (https://github.com/Tochemey/ModFreeSwitch). Now I want to have some directives because I am contemplating using Akka.IO to reimplement it. Thank you very much for your assistance.
Marc Piechura
@marcpiechura
Jul 27 2016 18:24
@Tochemey If you want to see more examples I have created a little project as learning exercise, https://github.com/Silv3rcircl3/Akka.Signal
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:25
@Silv3rcircl3 Thank you very much. I will check that too.
Aaron Stannard
@Aaronontheweb
Jul 27 2016 18:25
if you've already implemented that protocol in DotNetty
I would re-use it
no need to reinvent the wheel
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:26
@Aaronontheweb Ok
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:39
@Silv3rcircl3 Please which package holds the IImmutableMap?
I have pulled Akka package but still I am having an issue.
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:47
I have not gotten the IImmutableMap in the package?
Marc Piechura
@marcpiechura
Jul 27 2016 18:48
are you talking about Akka.Signal?
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:49
Yeah
Marc Piechura
@marcpiechura
Jul 27 2016 18:49
mh
will check it laiter
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 18:51
Ok I have pulled the code to grab the Akka.IO stuff.
to11mtm
@to11mtm
Jul 27 2016 19:16
@object : small world; I was trying to explain to a colleague Monday that activator is a bad thing, looked for an example and found a blog post you wrote. Thanks for making my life easier!
qwoz
@qwoz
Jul 27 2016 19:18
I've read https://petabridge.com/blog/akkacluster-state-distribution/ and one piece of feedback is that it's difficult to follow along because the /user/foo example is meaningless to me and I can't relate that to something practical. For example, if this were stated in terms of a banking application and the message contained "deposit to account 1234 in branch 789" it would help to see how something gets routed based on the account or branch number and how transactions around that account get distributed in the cluster. Would the router in that case be /user/branches and branch 789 might resolve to node 3, under which there's a child-per-entity actor at /user/branches/branch-789 which then processes the message, creating an actor for the account as /user/branches/branch-789/account-1234 beneath it?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 19:26
yep, that's about right
Arjen Smits
@Danthar
Jul 27 2016 19:26
@Tochemey nice project https://github.com/Tochemey/ModFreeSwitch. We use FreeSwitch as well. So its good to know its out there.
Aaron Stannard
@Aaronontheweb
Jul 27 2016 19:26
at a certain point for a large post like that where I'm already introducing a bunch of concepts at once I usually try to start limiting the scope of things
including the amount of detail in the examples
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 19:28
@Danthar Thanks your feedback is welcomed
Hello Can anyone point me to the API doc of ByteString?
Arjen Smits
@Danthar
Jul 27 2016 19:29
I plan on building an SIP Redirection server for an ACME packet
with Akka.IO
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 19:29
@Danthar I will be implementing the Akka.IO version of that library.
Arjen Smits
@Danthar
Jul 27 2016 19:30
Yeah you mentioned that. Im still in the research phase myself. And battling to get my project higher on the priorities list internally.
Peter Rosconi
@prosconi
Jul 27 2016 19:33
Is there an example of ClusterRouterGroup using RoundRobinGroup around?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 19:33
/myRouter {
  router = broadcast-group # routing strategy
  routees.paths = ["/user/foo"] # path of routee on each node
  cluster {
     enabled = on
     allow-local-routees = off
  }
}
change to
/myRouter {
  router = round-robin-group # routing strategy
  routees.paths = ["/user/foo"] # path of routee on each node
  cluster {
     enabled = on
     allow-local-routees = off
  }
}
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 19:43
Hello can someone please guide me through the usage of ByteString?
Marc Piechura
@marcpiechura
Jul 27 2016 19:45
@Tochemey I have updated Akka.Signal, it's now using Nuget again isntead of Paket and replaced the TreeMap with a simple Dictionary
the TreeMap was inside in Akka itself in version 1.0.6 but was removed since then
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 19:47
Ok
Thanks
How can I use IImmutableDictionary since it is part of Akka.Net?
Arjen Smits
@Danthar
Jul 27 2016 19:49
@Silv3rcircl3 do you have some small easy to use tool to generate animated gifs ? (aka what do you use?)
Marc Piechura
@marcpiechura
Jul 27 2016 19:49
@Danthar I used https://getsharex.com/ for it
Arjen Smits
@Danthar
Jul 27 2016 19:50
ah, thx!
Marc Piechura
@marcpiechura
Jul 27 2016 19:51
@Tochemey ImmutableDictionary isn't part of Akka.Net it comes with the System.Collections.Immutable package
and with what are you struggling regarding ByteString's ?
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 20:01
@Silv3rcircl3 Thank you for your assistance. However when you like ByteBuffer in DotNetty there is an API to understand it. I would like to know whether there is such one.
Marc Piechura
@marcpiechura
Jul 27 2016 20:05
I don't think that we have any sort of documentation in that area, but you can look at the source code as well as the specs, that should give you an overview
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 20:22
@Silv3rcircl3 is there any way to read the byte one after the other as if there is readerIndex mechanism?
Marc Piechura
@marcpiechura
Jul 27 2016 20:28
Sure, you can use it like a array with a for loop or use the ByteIterator or even IEnumerator
Arsene Tochemey GANDOTE
@Tochemey
Jul 27 2016 20:31
I have discovered that.
Aaron Stannard
@Aaronontheweb
Jul 27 2016 20:37
I think we have an N+1 error on the Broadcast<t> shape in Akka.Streams
hmm maybe not
issue is something else
public override IModule ReplaceShape(Shape shape)
        {
            if (!ReferenceEquals(shape, Shape))
            {
                if (!Shape.HasSamePortsAs(shape))
                    throw new ArgumentException("CombinedModule requires shape with same ports to replace",
                        nameof(shape));

                return new CompositeModule(SubModules, shape, Downstreams, Upstreams, MaterializedValueComputation,
                    Attributes);
            }
            return this;
        }
failing that check inside StreamLayout.cs
Aaron Stannard
@Aaronontheweb
Jul 27 2016 20:42
 _downloadGraph = GraphDsl.Create(builder =>
            {
                // html flows
                var downloadHtmlFlow = builder.Add(_downloadHtmlFlow);
                var downloadBroadcast = builder.Add(new Broadcast<DownloadHtmlResult>(2));
                var completedDownload = builder.Add(DownloadFlow.ProcessCompletedHtmlDownload());
                var parseCompletedDownload = builder.Add(ParseFlow.GetParseFlow(Job));
                builder.From(downloadHtmlFlow.Outlet).To(downloadBroadcast.In);
                builder.From(downloadBroadcast.Out(0)).To(completedDownload.Inlet);
                builder.From(downloadBroadcast.Out(1)).To(parseCompletedDownload.Inlet);

                // image flows
                var downloadImageFlow = builder.Add(_downloadImageFlow);

                var sourceBroadcast = builder.Add(new Broadcast<CrawlDocument>(2));
                builder.From(sourceBroadcast.Out(0)).To(downloadImageFlow.Inlet);
                builder.From(sourceBroadcast.Out(1)).To(downloadHtmlFlow.Inlet);


                return new SinkShape<CrawlDocument>(sourceBroadcast.In);
            });
graph construction
the ports are different (according to the error) but I can't tell where the shapes are being replaced
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:04
@Silv3rcircl3 looks like the issue is that the GraphDsl fails when I pass the inlet for the broadcast into the sink shape
what's the correct syntax for this?
Marc Piechura
@marcpiechura
Jul 27 2016 21:08
I think you haven't connected all ports
the ouput from parseCompletedDownload isn't connected
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:09
it doesn't produce any output
it's meant to be a sink
ah, I think I get it... I need to close all 3 of my branches in this graph don't I
Marc Piechura
@marcpiechura
Jul 27 2016 21:09
ah ok because ParseFlow.GetParseFlow(Job)) sounds like it would return a Flow
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:10
yeah, it's a flow
but the flow is meant to terminate into a sink with the way I wrote it earlier
let me check
might be missing something since I started porting this from flow
Marc Piechura
@marcpiechura
Jul 27 2016 21:10
so b.Add makes a copy of the shape you provide and then you need to connect all ports (Inlets and Outlets) of the result inside your graph
but you doesn't do that for the parseCompletedDownload
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:11
derp
yeah
forgot to add my sinks in here
have some actorref sinks that are the targets
Marc Piechura
@marcpiechura
Jul 27 2016 21:12
I see, yeah that should fix the problem
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:12
had them before when these were just flows
forgot to put them into the graph
Marc Piechura
@marcpiechura
Jul 27 2016 21:13
also you don't need to call .Inlet/.Outlet every time
b.From(broadcast).To(flow);
works just fine
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:14
I get compile time errors when I do that
Marc Piechura
@marcpiechura
Jul 27 2016 21:14
From knows it has to use the outlet of the shape and To knows it needs to use the Inlet
probably because you have different types for the Materialized value
source has mostly NotUsed while Sinks often have Task
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:15
yeah, it's the different TMats on that one
transformation flows
Marc Piechura
@marcpiechura
Jul 27 2016 21:17
you have two options for it
use something like this
Sink.Ignore<int>().MapMaterializedValue(_ => NotUsed.Instance)
this changes the TMat from Task to Notused for the sink
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:18
 _selfHtmlSink = Sink.ActorRef<CheckDocuments>(Self, PublishStatsTick.Instance);
            _selfDocSink = Sink.ActorRef<CompletedDocument>(Self, PublishStatsTick.Instance);
            _downloadHtmlFlow = Flow.Create<CrawlDocument>().Via(DownloadFlow.SelectDocType())
                .Buffer(10, OverflowStrategy.Backpressure)
                .Via(DownloadFlow.ProcessHtmlDownloadFor(DefaultMaxConcurrentDownloads, HttpClientFactory.GetClient()));

            _downloadImageFlow = Flow.Create<CrawlDocument>()
                .Via(DownloadFlow.SelectDocType())
                .Buffer(10, OverflowStrategy.Backpressure)
                .Via(DownloadFlow.ProcessImageDownloadFor(DefaultMaxConcurrentDownloads, HttpClientFactory.GetClient()))
                .Via(DownloadFlow.ProcessCompletedDownload());

            _downloadGraph = Sink.FromGraph(GraphDsl.Create(builder =>
            {
                // html flows
                var downloadHtmlFlow = builder.Add(_downloadHtmlFlow);
                var downloadBroadcast = builder.Add(new Broadcast<DownloadHtmlResult>(2));
                var completedDownload = builder.Add(DownloadFlow.ProcessCompletedHtmlDownload());
                var parseCompletedDownload = builder.Add(ParseFlow.GetParseFlow(Job));
                builder.From(downloadHtmlFlow).To(downloadBroadcast);
                builder.From(downloadBroadcast.Out(0)).To(completedDownload.Inlet);
                builder.From(downloadBroadcast.Out(1)).To(parseCompletedDownload.Inlet);
                builder.From(parseCompletedDownload).To(_selfHtmlSink);
                builder.From(completedDownload).To(_selfDocSink);

                // image flows
                var downloadImageFlow = builder.Add(_downloadImageFlow);
                builder.From(downloadImageFlow).To(_selfDocSink);

                var sourceBroadcast = builder.Add(new Broadcast<CrawlDocument>(2));
                builder.From(sourceBroadcast.Out(0)).To(downloadImageFlow.Inlet);
                builder.From(sourceBroadcast.Out(1)).To(downloadHtmlFlow.Inlet);


                return new SinkShape<CrawlDocument>(sourceBroadcast.In);
            }));
that worked perfectly
just had that parse petabridge.com
Marc Piechura
@marcpiechura
Jul 27 2016 21:18
great :)
have you added async boundaries for you buffers ?
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:19
that was actually my next question
how do those work?
I saw the method
but wasn't sure what all it did
Marc Piechura
@marcpiechura
Jul 27 2016 21:20
basically flows are fused together into one actor if you don't set async boundaries, for example 100 select statements would run in the same actor instead of 100 seperate ones
with .Async the part before runs in its own actor and the part after it
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:22
so on this graph up here
where would you apply that?
I guess in this case, I'd call an async boundary after the download part
allow the parsing to happen in separate actors
since that can benefit from parallelism
right?
Marc Piechura
@marcpiechura
Jul 27 2016 21:25
yeah after the buffer I guess
We already had an issue in that area with some more details
akkadotnet/akka.net#1922
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:27
those download flows look like this
  public static Flow<IDownloadDocument, DownloadImageResult, NotUsed> ProcessImageDownloadFor(
            int degreeOfParallelism, HttpClient client)
        {
            return Flow.Create<IDownloadDocument>()
                .Where(x => x is DownloadImage)
                .Select(x => (DownloadImage)x)
                .SelectAsyncUnordered(degreeOfParallelism,
                    document =>
                        client.GetByteArrayAsync(document.Document.DocumentUri)
                            .ContinueWith(DownloadImageContinuationFunction(document)));
        }
those are asynchronous and using PipeTo, yes?
so I figured that those didn't need to be marked as Async
since it already looks like it is
or am I mistaken there?
Marc Piechura
@marcpiechura
Jul 27 2016 21:29
ah yeah SelectAsyncUnordered also do the trick
so problem is if you block the thread as described in the issue the buffer is also blocked and can't consume from upstream, but since you use async stage the buffer can consume from upstream while these tasks are running
there is also a small part about it in the docs
-> Operator Fusion
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:34
the only blocking bit is the parsing at the moment
everything else is either super simple or async
Marc Piechura
@marcpiechura
Jul 27 2016 21:34
then it's good
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:36
have another fun issue I'm pulling up in a moment
where streams works great on one cluster node but not on a second as soon as I add it
get an avalanche of dead letters instead
Marc Piechura
@marcpiechura
Jul 27 2016 21:40
yeah dead letters is definitely a thing with streams, saw that in some tests too
but can't imagine why it would work in one node but not in another
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:44
lol my blog post on Akka.Cluster got removed from /r/csharp on reddit because it "wasn't related to C#" and was a "commercial advertisement"
how is it any less related to C# or any more commecial than Visual Studio Update 3 announcement from visualstudio.com on the homepage of that subreddit?
silly
Aaron Stannard
@Aaronontheweb
Jul 27 2016 21:53
I think I figured it out
some of the control messages being written back out were dropped due to max frame size
so it was Akka.Remote breaking the transparency of some of this stuff
I increased the setting by 128kb and that fixed it
Aaron Stannard
@Aaronontheweb
Jul 27 2016 22:03
blob
this is what I'm seeing a lot of @Silv3rcircl3
it's a 50/50 shot that it will work
versus eating itself alive with these dead letter messages forSubscribePending
my first node can do this work no problem
it's only the second node that explodes
but this stream is local-only
doesn't talk to any remote actors
[INFO][7/27/2016 10:03:08 PM][Thread 0007][akka://webcrawler/remote/akka.tcp/webcrawler@127.0.0.1:50902/user/api/https%3A%2F%2Fmsdn.microsoft.com%2F/coordinators/c2/StreamSupervisor-50/Flow-0-1-identityOp] Message SubscribePending from akka://webcrawler/remote/akka.tcp/webcrawler@127.0.0.1:50902/user/api/https%3A%2F%2Fmsdn.microsoft.com%2F/coordinators/c2 to akka://webcrawler/remote/akka.tcp/webcrawler@127.0.0.1:50902/user/api/https%3A%2F%2Fmsdn.microsoft.com%2F/coordinators/c2/StreamSupervisor-50/Flow-0-1-identityOp was not delivered. 7 dead letters encountered.
the stream never recovers
lies in a permanently failed state on that actor afterwards
and fails to process the additional messages it receives
Marc Piechura
@marcpiechura
Jul 27 2016 22:20
Are these actors deployed?
Looks like from the path
Aaron Stannard
@Aaronontheweb
Jul 27 2016 22:49
yeah, remotely deployed
they're routees on the other end of a cluster pool router