These are chat archives for akkadotnet/akka.net

16th
Feb 2018
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 11:24
just applied cluster singleton :) works OK, but when I kill the node on which the actor is running, it takes about a minute to spawn it on another node. Could it be related to the fact that I use Consul?
@Horusiath ^^
Bartosz Sypytkowski
@Horusiath
Feb 16 2018 12:34
@vasily-kirichenko It potentially could.
I don't remember right now if singleton migration is triggered when node becomes unreachable, or when it's marked as dead
but I'm almost sure that it's need to be identified as dead - unreachable nodes could happen in network partition quite often, so it would be bad to spawn another singleton as reaction for simple unreachability
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 12:39
the log is
12:47:29 INFO  [ClusterSingletonManager, 6] Previous oldest removed [akka.tcp://xxx@localhost:4888]
12:47:29 INFO  [ClusterSingletonManager, 6] Younger observed OldestChanged: [ -> myself]
12:47:29 INFO  [ClusterSingletonManager, 6] Singleton manager started singleton actor [akka://xxx]
12:47:29 INFO  [ClusterSingletonManager, 6] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
12:47:29 INFO  [0, Culture=neutral, PublicKeyToken=null]], 9] Getting server list...
12:47:30 ERROR [OneForOneStrategy, 11] Object reference not set to an instance of an object. Object reference not set to an instance of an object.
BTW, I have no idea how to catch the exception on the last line.
What can it be?
Bartosz Sypytkowski
@Horusiath
Feb 16 2018 12:57
No idea. Do you use some custom supervision strategy?
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 13:22
No.
Don't even specify one explicitly.
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 13:28
@Horusiath I have another question :) That cluster singleton actor spawns a couple of other actors and a stream. As far as I understand, it's its responsibility to shutdown the stream when it's stopping. Do you find the following code good or there is a better / simpler way to do it?
let stream mat =
    Source...
    |> Source.viaMat (KillSwitches.Single()) Keep.both
    |> Source...
    |> Source.``to`` Sink.ignore
    |> Graph.run mat

let props (mat: IMaterializer) : Props =
    props(
        let rec loop (streamKillSwitch: IKillSwitch) (msg: obj) =
            match msg with
            | LifecycleEvent PostStop -> 
                streamKillSwitch.Shutdown()
                ignored()

            | _ -> unhandled()

        fun (ctx: Actor<obj>) ->
            let taskQueue, streamKillSwitch = stream mat
            spawn ctx "another-actor" (Another.props taskQueue) |> ignore
            become (loop streamKillSwitch)
    ).ToProps()
Deniz ─░rgin
@Blind-Striker
Feb 16 2018 17:01
is it possible to define more than one dispatcher with different throughput to use in different actors. Like this ;
``` 
            custom-dispatcher-300 {
                type = Dispatcher
                throughput = 300
            }

            custom-dispatcher-400 {
                type = Dispatcher
                throughput = 400
            }

            custom-dispatcher-500 {
                type = Dispatcher
                throughput = 500
            }
``` 
Onur Gumus
@OnurGumus
Feb 16 2018 17:02
@vasily-kirichenko how would you gracefully handover a singleton ?
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:43
@OnurGumus Interesting question. The singleton mechanism seem to be based on cluster membership events, so a singleton is not moved until the node is unreachable (IMHO).
Onur Gumus
@OnurGumus
Feb 16 2018 17:43
I am thinking calling Cluster.Leave
however not sure how graceful it would be
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:44
but why do you need to move it to another node?
Onur Gumus
@OnurGumus
Feb 16 2018 17:45
The reason is I have two processes running. Ocassionally we want to upgrade the application. So we stop one, handover happens, upgrade the stopped, then start and stop the 2nd one.
So that we don't have downtime.
during the upgrade process.
I am not sure if this is the bestway though
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:45
so you want to switch fast? I dunno.
Onur Gumus
@OnurGumus
Feb 16 2018 17:46
Yes. I think this should be combined with a coordinated shutdown
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:47
I played with it today and it took about a minute :(
Onur Gumus
@OnurGumus
Feb 16 2018 17:47
No that's obviously wrong
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:48
The docs promise it should take "seconds"
Onur Gumus
@OnurGumus
Feb 16 2018 17:48
it is almost instant for me but do you use splitbrain ?
How do you down the other ?
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:48
I suspect the Consul was the issue
Onur Gumus
@OnurGumus
Feb 16 2018 17:48
Even if you close the application, you need to "down" it.
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:48
Ctrl+C :)
Onur Gumus
@OnurGumus
Feb 16 2018 17:49
that's not sufficient for handover
you also have to down it.
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:49
so the other nodes log that it's unreachable for that minute
Onur Gumus
@OnurGumus
Feb 16 2018 17:49
Either use auto down config or use split brain
Then it is normal
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:49
auto down is evil! evil!
:)
Onur Gumus
@OnurGumus
Feb 16 2018 17:49
Use split brain then.
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:49
how?
resolver you mean?
Onur Gumus
@OnurGumus
Feb 16 2018 17:50
It is very simple and documented
Yes
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:50
ah. will read about it.
Onur Gumus
@OnurGumus
Feb 16 2018 17:50
to tackle with network partitioning
Vasily Kirichenko
@vasily-kirichenko
Feb 16 2018 17:51
However, it's not very important in my case. The actor is just a batch worker, nobody even sends messages to it.
so a minute downtime is ok
Onur Gumus
@OnurGumus
Feb 16 2018 17:53
no it is not okay
you have to down it by some means
if you don't down it then I don't know maybe then it is okay
Bartosz Sypytkowski
@Horusiath
Feb 16 2018 20:28
@vasily-kirichenko kill switch invoked on PostStop seems fine.
@OnurGumus if you're able to tell cluster to leave gracefully i.e. via Cluster.LeaveAsync, it's always better - @vasily-kirichenko case regards non-graceful scenario, when the cluster must detect dead node instead of being explicitly informed about it leaving