These are chat archives for akkadotnet/akka.net

1st
Mar 2017
Chris G. Stevens
@cgstevens
Mar 01 2017 02:23
So if the elderst node is the singleton live on then what governs the role leader?
I hate when I can't edit.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:26
so if you specified a role for a singleton
the singleton will live on the oldest node of that role
which may not be the role leader
it's the Member.UpNumber that determines the age
Chris G. Stevens
@cgstevens
Mar 01 2017 02:26
What determines the role leader?
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:27
same sorting algorithm that determines the leader
but applied only to nodes of that role type
I can't remember exactly how it works
Chris G. Stevens
@cgstevens
Mar 01 2017 02:27
Kinda like the Cluster leader. I always think it should be another seed node but it isn't.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:27
the leader can change pretty easily
well now I'm curious
Chris G. Stevens
@cgstevens
Mar 01 2017 02:28
Yes. With my new service manger I can see it change
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:28
been a while since I've looked at this code
hang on a sec
Chris G. Stevens
@cgstevens
Mar 01 2017 02:28
Interesting to change my eye.
takes into account the reachability / status of the node
so only reachable up members can be the leader
and then uses the Ordering for all nodes that satisfy that criteria
Chris G. Stevens
@cgstevens
Mar 01 2017 02:30
Itr
there we go
so it compares all of the address parts
plus the UID
but the UID is the tie-breaker
not used normally
Chris G. Stevens
@cgstevens
Mar 01 2017 02:32
Ok cool. Thanks! I can answer that to some body who's asked me ;). Much appreciated
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:32
so we compare by Host name first using string ordering
and if that ends in a tie
then we compare by port
so yeah, a new node can join with an IP address or hostname that sorts "higher" than the current nodes
and that node can become leader once it's Up
Chris G. Stevens
@cgstevens
Mar 01 2017 02:33
Ohh
That is why
It goes to my service on my same host. It to my other seed node
Which is on my other server
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:34
yep, that could do it
it's not an expensive thing, to move the leader
Chris G. Stevens
@cgstevens
Mar 01 2017 02:34
Been wondering that for awhile
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:34
requires some convergence on the gossip in order to do it
which is how we ensure that there aren't two leaders in a cluster that isn't experiencing any reachability issues
if you have some nodes that are unreachable though
let's say in a split brain scenario
where the cluster gets split in half
you'll have two leaders until the partition clears
both the leaders won't take any action, by default, in terms of changing the cluster membership until the partition is resolved or the unreachable nodes are downed
although when we finish adding the WeaklyUp state that may change :p
Chris G. Stevens
@cgstevens
Mar 01 2017 02:36
:+1:
The have been restart my services but not my websites so hey become their own cluster. That is why I have added in a required Lighthouse to force my websites or services to restart and rejoin the Cluster.
They*
That is my only split experience so far.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:39
I've had issues with WebCrawler in the past where the CPU spiked to 100 and nodes started missing heartbeats
which is going to happen when machines get that busy
nodes became unreachable and work stopped getting distributed
gave the worker nodes time to catch up and they were able to rejoin the cluster again
also seen IIS do weird stuff
rebooted my web role right away at startup when SignalR turned on; happens about 40% of the time
Chris G. Stevens
@cgstevens
Mar 01 2017 02:40
Yea or my awaken Ninject memory leak.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:41
but the newer features we added to Akka.Cluster automatically downed the old incarnation of the node and replaced it with the rebooted one
Chris G. Stevens
@cgstevens
Mar 01 2017 02:41
Awesome. Oh my. Auto correct is killing me
Nice !
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:41
the new stuff we're going to be releasing soon
should make the graceful exit scenario much more robust
can automatically terminate the actor system when the process exits
automatically leave the cluster when the actor system gets terminated
and vice versa for those too
Chris G. Stevens
@cgstevens
Mar 01 2017 02:42
Nice. My next deployment allows me to roll out of my release and keep the Cluster up. :smile:
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:42
nice!
so you can replace the nodes in-place?
or bring in new nodes and decommission the old ones?
Chris G. Stevens
@cgstevens
Mar 01 2017 02:44
Yeah. I shut it down using teamcity and powershell and then deploy. Start it back up. Try my best for a clean terminate.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 02:45
nice, I did something similar with Elastic BeanStalk
used the AWS PowerShell commandlets to script new deployments
I've been really impressed with PowerShell; used it on a few projects recently
never used it much previously
Chris G. Stevens
@cgstevens
Mar 01 2017 02:46
Yeah use it right now just to start and stop services website and apppools
But the best thing I learned lately is using the recovery options on the services to allow the restart of the services, using application.exit and hostcontrol.stop. And the app domain.unload to restart the apppools when I need to rejoin the Cluster by doing a full restart.
The allows me to force a clean terminate and startup.
Rune Jacobsen
@havremunken
Mar 01 2017 12:24

Hi folks. I'm not even a n00b at akka.net, it just seems like an interesting fit for something I have to solve. Basically, I'm trying to create a bunch of processes that will each try to find a solution to a problem with so many combinations of possible outcomes that brute forcing it is unrealistic. Each process (actor in the akka.net world I guess) would bite off a small chunk and test it for feasability. They will hopefully be somewhat intelligent about the chunks they test.

However, testing a chunk that has already been tested is a waste of time. So we'll need a source of truth for which trunks have been tested, and which are still "available". I'm thinking some kind of root actor could handle this.

So a worker actor would ask the root actor for a chunk, and either get "ownership" of this chunk, or be told that this has already been tested.

Would this work well with akka.net, or would something else (TPL?) be a better fit for handling it? Performance is a feature here, since we're trying to figure out something that would normally require brute forcing of the entire problem space.

Natan Vivo
@nvivo
Mar 01 2017 14:36
@havremunken yes, that's a very common with akka
I use a similar pattern here to schedule work. you could do this with a router and the default mailbox, but if you want to track work and distribute more efficiently, this pattern of a root actor works very well
Rune Jacobsen
@havremunken
Mar 01 2017 14:38
@nvivo Excellent, thanks. So basically an actor should just ask for a chunk to work with, alternatively try to register for å specific chunk, and the "hub" would reply positively or negatively to the message? Sounds like I have a way forward. :)
Awesome, thanks. I'll make some time to understand some Akka examples. :)
Natan Vivo
@nvivo
Mar 01 2017 14:38
yes. with akka it's quite simple to create these mechanisms. it's quite natural to the framework
take a look, akka has quite a learning curve, but once you understand it, you won't be able to code anything without it
=)
Rune Jacobsen
@havremunken
Mar 01 2017 14:40
Hehe, sounds good! I understand some of the basic concepts (admittedly from hearing about them on .Net Rocks), but it is time to play around and see if I can get my head around it!
Natan Vivo
@nvivo
Mar 01 2017 15:11
@Aaronontheweb just saw your message regarding the CoordinatedShutdown
I think it's a good idea to retry a couple times if the socket is open, but I must say I think that after all, just disabling the Overlap Recycle is what is needed to make it work fine on asp.net
the downside is that the website is unavailable for a couple seconds between recycles, but it's not a huge issue. I have been testing this since yesterday, and even with the regular 20 minute idle recycle, I didn't see any problems
Maxim Cherednik
@maxcherednik
Mar 01 2017 15:15
@nvivo I guess we can also start actor system asynchronously with retry, can we ?
Natan Vivo
@nvivo
Mar 01 2017 15:15
That's an option too. I thought about doing that instead of just starting on Startup
But thinking about it, having the overlap recycle disabled makes perfect sense. since akka is a server, it doesn't make sense to have two of them running at the same time. the iis option is there for a reason
I wasn't very keen on using it at first, but it's not that bad
Maxim Cherednik
@maxcherednik
Mar 01 2017 15:18
those options are just to tweak around
Natan Vivo
@nvivo
Mar 01 2017 15:18
The only problem I see with the async start is that the website had other stuff running in the local actor system. if the actor system is delayed, that may be a problem
Unless there is a way to start the actor system without remoting at first, and start remoting later with a method call
Arjen Smits
@Danthar
Mar 01 2017 15:47

take a look, akka has quite a learning curve, but once you understand it, you won't be able to code anything without it

+1

@nvivo Thats not possible. The remoting capability is a complete actorprovider factory onto itself.
Replacing the actorrefprovider after starting and using your actor system would create numerous problems
Natan Vivo
@nvivo
Mar 01 2017 15:54
yeah, that's what I imagined
Sean Farrow
@SeanFarrow
Mar 01 2017 15:55
I'm creating new persistent plug-in, could someone tell me where the collection fixtures for the SQL plug-ins are implemented?
Guillermo
@mompox
Mar 01 2017 18:33
Quick question about Stash: when I unstash messages, is the original Sender preserved? Is not clear in the docs
Natan Vivo
@nvivo
Mar 01 2017 18:33
@mompox yes
Guillermo
@mompox
Mar 01 2017 18:34
@nvivo great, thanks
Andrew Young
@ayoung
Mar 01 2017 18:43
is there any documentation on how to use HOCON in my .NET proj outside of akka?
Jared Lobberecht
@Jared314
Mar 01 2017 18:45
Is there a form of MatchBuilder.Match() that accepts a MethodInfo as the message handler?
Natan Vivo
@nvivo
Mar 01 2017 18:53
@Jared314 what are you trying to do?
you can use ConfigurationFactoryto parse any file and then call the methods to extract the data. the basic usage is simple.
Andrew Young
@ayoung
Mar 01 2017 18:57
thanks.
@Jared314 i guess i wanted to be able to load a config from a section other than the default "akka" config section. but it seems that it is hard coded into the ConfigurationFactory.Load() method.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 18:59
yeah looks that way @ayoung
Andrew Young
@ayoung
Mar 01 2017 18:59
just created an issue to overload it.
Natan Vivo
@nvivo
Mar 01 2017 18:59
The Load is just a helper to get the settings from the default configuration places. it could be named LoadDefault
Andrew Young
@ayoung
Mar 01 2017 19:00
@Aaronontheweb would you entertain a PR for that?
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:00
yep, as long is it didn't break any existing methods
if it's just a new overload then that's 100% a-ok
Andrew Young
@ayoung
Mar 01 2017 19:01
doing it.
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:02
I have some frustrating race conditions with the CoordinatedShutdown PR I need to fix before we do our next release
have one unit test that fails due to the TestProbe that expects a MemberRemoved event being killed by the ActorSystem.Terminate call that occurs as part of the ordered shutdown (so the message gets DeadLetter'd instead)
and a new Cluster.Sharding spec that fails for reasons unknown at the moment sometimes
so if your PR is easy to review we can pack it into this release probably
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:08
Hi @Aaronontheweb, where can I read about the CoordinatedShutdown? I thought we already kinda have it and there was a racy thing about it...
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:09
akkadotnet/akka.net#2516 - it was implemented as a JVM feature in Akka in January
you can read their original PR notes in there
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:09
I thought the technic I was doing about leaving the cluster and then shutdown the system is enough
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:09
CoordinatedShutdown automates that process behind the scenes
you don't have to do anything
other than setting some HOCON config defaults
Natan Vivo
@nvivo
Mar 01 2017 19:09
how do you call it?
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:09
you can trigger CoordinatedShutdown in one of three ways
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:10
ok, but that was just the waiting part...
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:10
calling ActorSystem.Terminate
exiting the process
or downing / leaving a node in the cluster
you can disable / enable some of those via HOCON
Natan Vivo
@nvivo
Mar 01 2017 19:10
you got me on Terminate =)
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:11
I'll be adding a document that describes that to the website as part of the release
it's one of my outstanding to-dos on the PR
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:11
I see
I was just wondering that part about the clr hook....
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:11
the CLR hook is specifically for the process exit scenario
but yeah, there's an opportunity to run other cleanup tasks as part of that
Natan Vivo
@nvivo
Mar 01 2017 19:12
I honestly think trying to add a clr hook is too much
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:12
which is what you're referring to I think
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:12
isn't a bit overcomplication?
Natan Vivo
@nvivo
Mar 01 2017 19:12
it's better to depend on the user calling Terminate somewhere
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:12
I mean, user supposed to do graceful shutdown himself anyway
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:13
the thing I'm on the fence about with that issue
is having a way to force the ActorSystem to clean itself up if the process is exited isn't an altogether bad thing
even though that's not how you should do it
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:14
of course, if it's cheap to implement
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:14
it's definitely not expensive
Maxim Cherednik
@maxcherednik
Mar 01 2017 19:14
and it doesn't overlap with anything else
ok then :)
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:14
just requires hooking into the AppDomain.ProcessExited event
which is still supported on .NET Core
Natan Vivo
@nvivo
Mar 01 2017 19:14
I think it's too much magic for akka.net
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:15
it's disabled by default
I think
Natan Vivo
@nvivo
Mar 01 2017 19:15
what about aspnet?
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:15
coordinated-shutdown {
    # The timeout that will be used for a phase if not specified with 
    # 'timeout' in the phase
    default-phase-timeout = 5 s

    # Terminate the ActorSystem in the last phase actor-system-terminate.
    terminate-actor-system = on

    # Exit the CLR(Environment.Exit(0)) in the last phase actor-system-terminate
    # if this is set to 'on'. It is done after termination of the 
    # ActorSystem if terminate-actor-system=on, otherwise it is done 
    # immediately when the last phase is reached.  
    exit-clr = off

    # Run the coordinated shutdown when the CLR process exits, e.g.
    # via kill SIGTERM signal (SIGINT ctrl-c doesn't work).
    run-by-clr-shutdown-hook = on
so the CLR shutdown hook will run automatically by default
the exit-clr option will kill the process itself
once the ActorSystem terminates
Natan Vivo
@nvivo
Mar 01 2017 19:17
i was going to ask that
why would that be part of akka?
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:17
and that's the one that is off by default
if Akka is your app
as it is for most of the Windows Services that run it
being able to run the shutdown process in one shot might be appealing
and in general, most end-user code I've seen doesn't do stuff like have the ActorSystem.Terminate call hook into shutdown events exposed by other frameworks (even though you should)
Natan Vivo
@nvivo
Mar 01 2017 19:18
i think it's trying to do too much. akka doesn't even dispose my schedules when an actor stops... =)
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:18
they just bluntly shut it down and then wonder why their cluster doesn't work
Natan Vivo
@nvivo
Mar 01 2017 19:19
but that's just me
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:19
it's not really too much, just a phase in the shutdown process
personally, it was pretty cool seeing this stuff get run when I left it on in the multi-node test runner
could see that it worked
Natan Vivo
@nvivo
Mar 01 2017 19:20
I mean, I agree with run-by-clr-shutdown-hook, it's the exit-clr that sounds strange to be part of akka
yeah, but let's see
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:20
yeah that's why it's off by default
it would have side-effects
if it was left on by default
should be something the end-user opts into
personally, I don't think I'll be using it in my apps
in ASP.NET I call ActorSystem.Terminate inside the global.asax shutdown method
whose name I forget at the moment
and in Topshelf they have a service stopping method I hook into
I'd prefer to stick with those options because it's obvious to someone who's familiar with those more widely known tools but not Akka.NET what's happening and when
the other thing that's cool about CoordinatedShutdown though\
is you can hook in your own tasks on each shutdown stage
so say for instance you pipe some of your data over a ZeroMQ connection
and you need to shut that down before you finish leaving the cluster
you can hook into the CoordinatedShutdown.PhaseClusterLeave stage and register a task that can do that
all of the tasks for a particular stage are run in parallel for that stage before moving onto the next one
and you can add new stages too and declare those via HOCON
so what's nice about this plugin is that it also helps do things like make ClusterSingleton and Cluster.Sharding hand-over more regulated
Aaron Stannard
@Aaronontheweb
Mar 01 2017 19:25
and that's the part I'm still mostly working on right now are those bits
from an end-user's perspective, it's unlikely you'll ever need to call CoordinatedShutdown directly
but if you want to, now you can
Andrew Young
@ayoung
Mar 01 2017 21:18
how do you iterate through the properties of a HOCON object?
or get a list of the property keys
Natan Vivo
@nvivo
Mar 01 2017 21:21
The is an AsEnumerable() method on the config object
Andrew Young
@ayoung
Mar 01 2017 21:23
schweet. thanks.
Christian Duhard
@cduhard
Mar 01 2017 21:32
is there any information out there on the performance of various DB's for Akka.Persistence?
for instance I am curious how Cassandra performs vs SQL Server. I have a live production Akka.net system using EventStore
Natan Vivo
@nvivo
Mar 01 2017 23:34
Is it possible to manually create an ActorSelection with a list of paths?