These are chat archives for akkadotnet/akka.net

16th
Dec 2016
Chris Ochs
@gamemachine
Dec 16 2016 00:01
akk logs on the game node that it's starting singleton actor
Chris Ochs
@gamemachine
Dec 16 2016 01:08
ya I must not be configuring roles correctly, because the cluster thinks lighthouse is the leader
and the singleton manager never starts on any node
actually the actor starts, but I never get the singleton identified message, and it can't be reached via the proxy
Chris Ochs
@gamemachine
Dec 16 2016 01:17
so console logs show the nodes in the right role. So I'm thinking this is a bug or there is some undocumented something else here I need to change
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:25
hmm
so on all of the nodes where you need a singelton, you're staring up the ClusterSingleton plugin by getting a proxy or starting the real actor itself, right?
having lighthouse be the leader shouldn't have an impact on the cluster singleton if there are role settings enabled
Chris Ochs
@gamemachine
Dec 16 2016 01:36
ya I've been using the singleton for a while now. I just did another test where I start seed first, then game node. singleton actor is not identified. If I kill the seed node, the game node say's it's downing itself, also removes the seed node, then it identifies the singleton manager. It seems like the logic for oldest node is just not using roles correctly
I'm starting the singleton and proxy via code
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:37
It seems like the logic for oldest node is just not using roles correctly
aha
could you write that up into an issue?
that should be very easy for us to test against
Chris Ochs
@gamemachine
Dec 16 2016 01:37
ya np
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:39
we've been looking into a number of bugs in that module, mostly around ClusterClient, so stuff like this is invaluable to us
this type of information
Chris Ochs
@gamemachine
Dec 16 2016 01:42
if it's at all iffy I might just not use a singleton here, go to a global lock on the db which in this case is workable. As I've got basically 3 weeks to a launch with several hundred thousand in marketing and our use of the singleton being a core feature
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:43
if it's easy to reproduce we should be able to roll out a patch for it quickly
Chris Ochs
@gamemachine
Dec 16 2016 01:43
ok let me write it up here foryou
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:43
going to be staging one anyway soon
Chris Ochs
@gamemachine
Dec 16 2016 01:53
ok created issue #2415
Aaron Stannard
@Aaronontheweb
Dec 16 2016 01:54
excellent, thank you @gamemachine
// build queue is paused until further notice, working on upgrading our VMs to include the tools necessary for SandCastle docs and pre-loading them with some docker images used in our Akka.Persistence CI process
Aaron Stannard
@Aaronontheweb
Dec 16 2016 03:10
// builds are running again
Aaron Stannard
@Aaronontheweb
Dec 16 2016 03:22
going to be super exciting once we have akkadotnet/Akka.Persistence.SqlServer#54 passing, just have a minor issue with the Docker daemon running on Windows Server 2016
TL;DR; we're going to have the ability to integration test all of the officially supported Akka.Persistence plugins against one or more versions of the database they target using Docker on both Linux and Windows for this
we've been working on this behind the scenes for months now; needed to wait until Win2016 RTMed and some of the standard stuff like SQL Server express was dockerized
we've had that PR pass once already, but not yet on our auto-scaling agents... so almost there
Arjen Smits
@Danthar
Dec 16 2016 09:53
Hmm note to self. Dont do this:
  public MyActor()
        {
            Receive<Tick>(t =>
            {
                MethodThatThrowsException();
            });

            Context.System.Scheduler.ScheduleTellRepeatedly(TimeSpan.FromMilliseconds(500), TimeSpan.FromMinutes(1), Self, Tick.Instance, Self);
        }
because the scheduler registration is not cleared when the actor restarts
So if you dont cancel it yourself :P you end up spamming yourself to death
Bartosz Sypytkowski
@Horusiath
Dec 16 2016 09:53
@Danthar note on the first place - never use ScheduleTellRepeatedly :P
Arjen Smits
@Danthar
Dec 16 2016 09:54
yeah that as well :D
just ran into an issue in testing, where this type of code caused an OOM
because it kept failing, and the failrate was growing exponentially
because at one point, it got 500 Tick messages per minute instead of 1
Sergey Prytkov
@Rattenkrieg
Dec 16 2016 10:25
Guys can someone help me to understand this failure:
everything is green on my machine
Arjen Smits
@Danthar
Dec 16 2016 13:01
@Rattenkrieg the MNTK is abit racy. Looks like UnreachableNodeJoinsAgainSpecMultiNode4.AClusterOf4MembersMust is failing. I restarted the run to see if its racy.
Sergey Prytkov
@Rattenkrieg
Dec 16 2016 13:16
@Danthar thanks, hope to see passing tests
But regarding the output in TeamCity - I can't see any concrete name of failing test, how you managed to figure it?
Kris Schepers
@schepersk
Dec 16 2016 13:37
Hi, is there a good resource on how to write unit tests for persistent actors?
Kris Schepers
@schepersk
Dec 16 2016 13:49
E.g. is it possible to recover an actor given a list of events from a dummy journal?
Bartosz Sypytkowski
@Horusiath
Dec 16 2016 14:21
@schepersk sure, the only problem here is that dummy journal - I don't think, we have any ready one prepared right now
Kris Schepers
@schepersk
Dec 16 2016 14:25
@Horusiath do you have an example of a test using an inmem journal or a 'real' journal?
Bartosz Sypytkowski
@Horusiath
Dec 16 2016 14:28
I think you could take a look at Akka.Persistence.Tests project - it's a test suite for akka persistence itself.
Arjen Smits
@Danthar
Dec 16 2016 14:37
@Rattenkrieg i know abit about how the MNTK outputs its errors
although actually diagnosing what the real problem is can be very hard
searching in the build log for FAIL finds the failing test
Sergey Prytkov
@Rattenkrieg
Dec 16 2016 14:41
@Danthar got it, thanks. Now tests are passing, so it have to been racy issue.
Kris Schepers
@schepersk
Dec 16 2016 14:47
@Horusiath owkay, thanks. Will take a look
hidavidpeng
@hidavidpeng
Dec 16 2016 14:56
hi @all Does Akka.Net support .net core?
Arsene T. Gandote
@Tochemey
Dec 16 2016 14:57
Per the information I got it will be around in 1.5
Martin Helmer
@helmerm
Dec 16 2016 15:47
Hi. What is the proper way to shut down the ActorSystem in a WPF application. The Terminate() Task does not finish, even if I wait for it on a newly spawned background thread.
Martin Helmer
@helmerm
Dec 16 2016 15:52
I think my actors using the WPF dispatchers can not shut down, if the Dispatcher is already stopped
Arjen Smits
@Danthar
Dec 16 2016 15:55
@helmerm that sounds strange. Can you create an gist that reproduces the problem and create an issue on github for it ?
Martin Helmer
@helmerm
Dec 16 2016 15:58
@Danthar ok
Martin Helmer
@helmerm
Dec 16 2016 16:32
@Danthar I am using Caliburn.Micro. Plain WPF works just fine. akkadotnet/akka.net#2416
Aaron Stannard
@Aaronontheweb
Dec 16 2016 17:53
@Rattenkrieg looks like an intermittent MNTR failure
with a clustering spec
sometimes that can happen - I'll re-run it
Aaron Stannard
@Aaronontheweb
Dec 16 2016 17:59
@helmerm > WPF dispatchers can not shut down
that is an interesting edge case
you might be right
we fixed all sorts of behavior in Akka.NET 1.1 to guarantee that all actors have terminated in the correct order before we destroy the ActorSystem
but I introduced a bug in the dispatcher system that caused the SynchronizedDispatcher not to start correctly - fixed that in 1.1
err, 1.1.1 :p
strong possibility I didn't do the same for checking to see if it terminates correctly
@Horusiath @Danthar schedule tell repeatedly is fine :p , but you should always call it OnPreStart
Sergey Prytkov
@Rattenkrieg
Dec 16 2016 18:14
@Aaronontheweb seems that kind of tests are sensitive to hardware it runs on, have never expirenced such issues on my machine. Btw it's already been re-runed and finished successfuly.
Aaron Stannard
@Aaronontheweb
Dec 16 2016 20:04
we run those builds on intentionally underpowered machines
qwoz
@qwoz
Dec 16 2016 20:45
@Horusiath about "never use ScheduleTellRepeatedly" -- was that meant tongue-in-cheek, or is that because you want the actor to receive a non-repeated scheduled tell, act upon it, then schedule the next tell itself?
Aaron Stannard
@Aaronontheweb
Dec 16 2016 20:49
@qwoz I think he meant it tongue in cheek :p
qwoz
@qwoz
Dec 16 2016 20:55
ok, good to know! I use it several places myself and was wondering if I should make a few changes. Though the behavior where repeated tells survive an actor restart wasn't something I had thought of. What's the best practice there? Cancel any existing ones in PostRestart or similar?
Aaron Stannard
@Aaronontheweb
Dec 16 2016 20:57
@qwoz yeah, exactly
always cancel the task on PostStop
which is called by PreRestart when an actor restarts
assuming you call base.PreRestart
qwoz
@qwoz
Dec 16 2016 20:59
makes sense
Chris Ochs
@gamemachine
Dec 16 2016 23:31
Hey anyone have feedback on latency issues running in azure? We have a realtime multiplayer game. It's mobile not an mmo or fps. Normally for latency sensitive things I keep everything I can local, clients connect to the node a zone/game is on. But here I'm thinking about just routing messages to and from the nodes players are on to the node the game is running on. IT removes some complexity in our load balancing to do that. But I'm afraid of latency, with most cloud networks geared for http, and azure being relatively new at this and all.
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:31
good question
I've had issues with Azure Cloud Services not managing sockets well, but that's legacy stuff that Azure is trying to deprecate
truth is that with ARM that the virtual networks that run inside Azure are reasonably good by cloud standards
Chris Ochs
@gamemachine
Dec 16 2016 23:33
ya here 10ms here or there is not a huge deal, unlike some other stuff I've worked on where we are sending things like location updates to the client every 50ms
ya I'm pretty new to azure, guess I just need to do some testing
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:33
I've not run the same sized workloads I have with AWS
on Azure as of yet
Chris Ochs
@gamemachine
Dec 16 2016 23:34
ya most of my experience is with aws, or actually I usually avoid clouds for realtime games, but don't have a choice this time around
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:34
my previous company, MarkedUp, ran Akka.NET + Remoting along with big HTTP, Cassandra, Hadoop, and Solr workloads
the marketing automation piece, which was what we built with Akka.NET had, worked within < 1.5s real time constraints for elastic workloads
so I've had positive experiences there, and that was over 3 years ago
err, 2 years ago
ARM supposedly has better benchmarks on a lot of that stuff over AWS, but I've never measured it
usually we outperformed our 1.5s real time constraint by a long shot.... most of the latency there was pushing a notification over the network to the client
the Akka.NET part was executing all of that stuff in 20-30ms
Chris Ochs
@gamemachine
Dec 16 2016 23:38
ya I have a project for large scale mmo type games on java akka, with restraints on latency of around 5ms at around 400k messages second per node. Biggest challenge is really GC once you start working on low latency stuff
throw in an average cloud vm with performance that's anything but deterministic, and things can go south really fast
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:39
noisy neighbor problem on cloud VMs is my favorite
colleague of mine built a real-time customer support service that had to work across cellular networks, ran everything on Amazon originally
noticed that none of their VMs were hitting the numbers they were supposed to given their size
started measuring this metric amazon tracked called "clock steal"
Chris Ochs
@gamemachine
Dec 16 2016 23:40
ya last large game I worked on we planned to use vm's, then two weeks before launch after our testing showed how noisy the neighborhood really was, went with bare metal
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:40
TL;DR; turns out they were running on the same physical hardware as FourSquare at the height of its popularity, and the CPUs they were using were actually still being mostly utilized by them
Chris Ochs
@gamemachine
Dec 16 2016 23:41
although azure actually does have some of the best high end vm's from what I can tell
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:41
ended up moving to Joyent eventually
Chris Ochs
@gamemachine
Dec 16 2016 23:42
ya when it's my own choice I have a small list of lesser known but still large enough not to fail providers that give me a lot more for the money then big name cloud providers do
it's really the big guys that oversell the worst
because it just takes one no nothing manager in the right position that won't touch anything but a brand name
and they know that
Aaron Stannard
@Aaronontheweb
Dec 16 2016 23:43
yeah
the tooling ecosystems around some of those vendors is a big part too
Azure's invested a lot of money into getting integrated into things like HashiCorp tools, partnering with companies like DataStax, etc