These are chat archives for akkadotnet/akka.net

13th
Aug 2015
John Haigh
@haighis
Aug 13 2015 01:57
@rogeralsing or @Aaronontheweb i have a PR for review: akkadotnet/HOCON#23
billyxing
@billyxing
Aug 13 2015 07:41
Hello, I need some helps about Handling Network Failures. in my test project, there are a server and several clients. they are connected using akka remote. When a client is shutdown, the server will print out akka.remote.endpointdisassociatedException: Disassociated. So what is the right approach to handle network failures. I tried the deathwatch, but the terminated message always arrived after the exception and raised that exception one more time.
Tomas Vana
@creepone
Aug 13 2015 11:18
I just found out that if a remote actor system is shortly down and then comes up again (e.g. on a service restart), I get no Terminated messages when watching actors on this system. Is this by design ? If so, is there a way to find out that the actor system was restarted / remoting node disassociated (basically same as @billyxing question) ?
Anthony Brown
@bruinbrown
Aug 13 2015 11:53
Are there any notes on workflow when adding protobuf based stuff anywhere? E.g. existing tooling, where to put .proto files etc
Damian Reeves
@DamianReeves
Aug 13 2015 12:34
Does a PersistantActor also persist its unprocessed messages?
Also does it persist stashed messages?
Bartosz Sypytkowski
@Horusiath
Aug 13 2015 15:54
@DamianReeves no and no :) This is basically what AtLeastOnceDelivery is for. Persisting messages in mailbox themselfs is not a solution because it doesn't ensure that message (even received one) has been correctly processed i.e. actor didn't die during message processing
right now I'm a little occupied, but this pattern definitely deserve some example and more attention
Christian Sparre
@christiansparre
Aug 13 2015 16:08
Any good tips on how to organize internal state of actors. I mean do you use the same approach or very different ones depending on the actor?
And for debugging do you implement ways of getting a "snapshot" of the state, like sending a message "Give me a snapshot of your state"?
Roger Johansson
@rogeralsing
Aug 13 2015 16:12
clustervis2.png
Christian Palmstierna
@cpx86
Aug 13 2015 16:48
I need some help with an unstable remoting test. Any experts around that could provide ideas/insights?
The test is RemotingSpec.Drop_sent_messages_over_payload_size. It expects to receive exactly one OversizedPayloadException, but in some cases, it receives more. When it's the only test run, and it's run once, it always passes when I run it locally. If I run it multiple times in a row, it sometimes fails because it receives two messages instead of one. If I put the test code in a loop, it always fails - usually it receives between 2-8 messages.
What could be causing the exception to be logged multiple times? It seems like it is processing the message multiple times. Is there some retry action involved here that I am missing?
Ivan R. Perez
@irperez
Aug 13 2015 18:24
@Aaronontheweb I could really use this pull request currently. (petabridge/akka-monitoring#13)
@Horusiath +1 for a AtLeastOnceDelivery sample
Rodrigo Vidal
@rodrigovidal
Aug 13 2015 18:28
@rogeralsing this is really cool! where is the source code for this? (sorry I have'nt been following the project recently)
Aaron Stannard
@Aaronontheweb
Aug 13 2015 19:11
@irperez ah, thanks for bringing that to my attention - saw that we ran a new build of it earlier this week
ah man, @MaciekLesiczka did an awesome job with the readme... even put an ANIMATED GIF of the performance counters from Akka.NET working
I'll do a quick update on Akka.Monitoring and release an updated NuGet package
@cpx I'll help you with that! just cloned your fork to my local machine yesterday and was going to troubleshoot it some before the virtual meetup
I'll take a stab at it in the next couple of hours here
Maciek Lesiczka
@MaciekLesiczka
Aug 13 2015 19:20
hi there:)
@Aaronontheweb thanks for merge
my 1st PR in akka.net, although not in akka.net core repo:)
btw is it posibile to build akka.net in net 4.0, not 4.5?
Aaron Stannard
@Aaronontheweb
Aug 13 2015 19:22
@MaciekLesiczka hey, it's still an important contribution! especially if other people (@irperez) are pinging me and telling me I need to release it :p
Arjen Smits
@Danthar
Aug 13 2015 19:22
we get that question frequently
Maciek Lesiczka
@MaciekLesiczka
Aug 13 2015 19:26
I guess not
Aaron Stannard
@Aaronontheweb
Aug 13 2015 19:27
it's possible, but you'd have to write a number of #ifdef macros and such to do it
to strip out some of the async code we use
but the performance of Akka.NET on .NET 4.0 might be a lot lower, due to all of the TPL improvements in .NET 4.5
Aaron Stannard
@Aaronontheweb
Aug 13 2015 19:50
just pushed it
@bruinbrown unfortunately not - we don't have to do that very often so it hasn't been documented. Last time I had to do this was for Akka.Cluster
now that I think about it, we should include the original .proto files here https://github.com/akkadotnet/akka.net/tree/dev/src/core/Akka.Cluster/Proto if we don't already
for clustering at least
the codegen tools for creating C# files based off of a .proto description can be found here: https://github.com/google/protobuf/tree/master/csharp
I don't think we do anything special in particular, other than always make sure the generated code ends up in its own internal namespace
@creepone you should get Terminated messages if any of the following are true:
  1. the sockets were shut down
Aaron Stannard
@Aaronontheweb
Aug 13 2015 19:55
  1. it took longer than the deadline failure detector's breach threshold for the system to come up
you have to explicitly call Context.Watch or you have to remotely deploy an actor onto that system to get a termination notification
upgrade to 1.0.5 immediately
it resolves an issue where sockets were not cleanly shutdown in Akka.Remote
but on the flipside, if you just nuke the process (i.e. you don't actually program your windows service to call System.ShutDown())
the sockets might still linger open for a brief period of time
so make sure your Windows service performs a shutdown call when it stops
and upgrade to the latest
@billyxing > When a client is shutdown, the server will print out akka.remote.endpointdisassociatedException: Disassociated
I want to you make the following distinction for me
exception != something is wrong
it means something exceptional happened
Akka.Remote will always throw a disassociationexception and mark it as an error, even if it's an intentional planned termination
reason for this is so it always shows up in the logs
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:01
I get that question a lot - and the answer is just to not make it mean something that it isn't
Anthony Brown
@bruinbrown
Aug 13 2015 20:01
@Aaronontheweb awesome, I've managed to work around it now but I'll move stuff around when I refactor. Hopefully I'll be able to get a PR in soon for the list data stuff
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:02
we have an outstanding issue to make disassociation messages clearer specifically akkadotnet/akka.net#1087
Anthony Brown
@bruinbrown
Aug 13 2015 20:03
@Aaronontheweb I was planning on doing a partial pull request for now, since I've only implemented a couple of the CRDTs in C# so far, it might make a decent up for grabs issue
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:03
when you should care about disassociations is when they happen unexpectedly
When a client is shutdown, the server will print out akka.remote.endpointdisassociatedException: Disassociated
in this case, since the error message is "disassociated"
this actually is an error - it means an unplanned termination
if you shutdown a client and make a call to System.Shutdown() first
that log will change to akka.remote.endpointdisassociatedException: Shutdown
which is benign
@bruinbrown cool, we might have to create a feature branch for that - but we'll discuss that on the PR!
looking forward to it
Anthony Brown
@bruinbrown
Aug 13 2015 20:08
@Aaronontheweb it's almost complete as is, just got a few bits left to finish, btw any thoughts on the monotonic clock issue from the other day? Happy to make the change, I'd just be interested in a little guidance since it might touch a lot of clustering code
Ivan R. Perez
@irperez
Aug 13 2015 20:08
@Aaronontheweb thanks for merging the pull request. @MaciekLesiczka thanks for the new feature!!!
This message was deleted
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:08
@bruinbrown is there any reason why it should be made public? otherwise you can just expose it via a friend assembly
Properties/AssemblyInfo.cs inside the core Akka.NET project will show you where we did that in a number of areas
that'll give your assembly access to the MonotonicClock even though it's internal
Anthony Brown
@bruinbrown
Aug 13 2015 20:10
@Aaronontheweb cool, I'll take that approach
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:14
@billyxing @creepone we'll make sure that #1087 gets done - it's important to make the reason for those messages clearer. But in the meantime, remote deathwatch should work fine. Could you show me a gist where you're setting it up and handling it?
@cpx taking a look at your PR now
going to see if I can replicate that error
yep, repro'd it
Aaron Stannard
@Aaronontheweb
Aug 13 2015 20:43
looks like the message is being sent twice...
interessant
Aaron Stannard
@Aaronontheweb
Aug 13 2015 21:13
ha! I figured it out
@cpx so one of the things we do inside Akka.Remote when we first create an association is we put all non-system messages into a buffer, queued for delivery after the association opens
in the event that some of those buffered messages fail delivery, we schedule a backoff + retry
so the reason why we're seeing two OversizedPayloadExceptions is because of the retry
this is the culprit:
private void Writing(object message)
        {
            if (message is EndpointManager.Send)
            {
                var s = message as EndpointManager.Send;
                if (!WriteSend(s)) //returns false on oversized payload
                {
                    if (s.Seq == null) EnqueueInBuffer(s);
                    ScheduleBackoffTimer();
                    Context.Become(Buffering);
                }
            }
inside Endpoint.cs
Aaron Stannard
@Aaronontheweb
Aug 13 2015 21:22
ok, this is an easy fix actually
I'll post a comment on the diff
John Haigh
@haighis
Aug 13 2015 23:23
@Aaronontheweb akkadotnet/HOCON#11