These are chat archives for akkadotnet/akka.net

14th
Aug 2016
JoeCoJabba
@joecojabba
Aug 14 2016 02:11
Hi, does anyone have any tips on configuring akka remote on a azure vm. I believe I have the VM firewall wide open atm. I can rdp and hit webserver; but I get a timeout on my ActorSelection in the akka client app.
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:12
@joecojabba do you have exposed ports on your Azure security group that can reach the Akka.NET application?
Windows VMs have to have their own built-in Windows Firewall ports opened as well as the Azure-level groups
JoeCoJabba
@joecojabba
Aug 14 2016 02:13
hmm... well I have did open the floodgates at the azure level and disabled the Windows Firewall
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:14
floodgates meaning you opened all ports?
JoeCoJabba
@joecojabba
Aug 14 2016 02:14
I was able to hit IIS default page for example
yes
I've tried combinations of localhost/public-hostname/0.0.0.0/
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:14
could you post your akka.remote configuration?
JoeCoJabba
@joecojabba
Aug 14 2016 02:14
sure
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:14
and which version of Akka.NET?
1.1.1?
JoeCoJabba
@joecojabba
Aug 14 2016 02:15
yes 1.1.1
 <hocon>
      <![CDATA[
          akka {  
                    actor{
                        provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote",
                        serializers {
                            wire = "Akka.Serialization.WireSerializer, Akka.Serialization.Wire"
                        }
                        serialization-bindings {
                            "System.Object" = wire
                        }                    

                    }
                    remote {
                        helios.tcp {
                            port = 8090
                        hostname = 0.0.0.0
                public-hostname=65.x.x.x

                        }
                    }
                }
      ]]>
    </hocon>
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:17
ok, and when your Akka.Remote application boots up - does it indicate that the remoting system was bound successfully?
it's an INFO-level log
displays the bound address
and what do your logs say on the client when you attempt to connect?
JoeCoJabba
@joecojabba
Aug 14 2016 02:20
server is
[INFO][8/14/2016 2:19:04 AM][Thread 0001][remoting] Remoting started; listening
on addresses : [akka.tcp://DeployTarget@65.52.244.82:8090]
[INFO][8/14/2016 2:19:04 AM][Thread 0001][remoting] Remoting now listens on addr
esses: [akka.tcp://DeployTarget@65.52.244.82:8090]
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:20
k, that looks good
JoeCoJabba
@joecojabba
Aug 14 2016 02:21
so client should be pointing to public ip yeah?
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:21
yep - client should be able to do that just fine
JoeCoJabba
@joecojabba
Aug 14 2016 02:21
what about a client running on the same machine?
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:22
as long as the IP system resolves to a real Akka.NET application everything should work
one catch
Akka.NET is strict, for good reasons, about the address used to reach it; if you had a machine running locally try to connect to that process using localhost
that would insta-fail since the public-hostname was already set to 65.**
Akka.Remote would receive the packets and deserialize them
since that listed address didn't match the bound address
and then drop it
that's designed to guarantee that everyone is talking about the same actors
so why you're not able to reach the nodes - definitely check the logs
check to see if the server allows an association with the client
and vice-versa
and check for deadletters on the server too
might mean that your client is successfully connecting but isn't selecting a live actor
so the message gets dumped into dead letters since the server can't find anyone to deliver it to
JoeCoJabba
@joecojabba
Aug 14 2016 02:27
hmm... k...
well heres's the client output from same within same VM
Remoting now listens on addresses: [akka.tcp://Deployer@0.0.0.0:49231]
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:29
have any logs for the actual message sends?
JoeCoJabba
@joecojabba
Aug 14 2016 02:29
[WARNING][8/14/2016 2:24:28 AM][Thread 0007][remoting] Tried to associate with unreachable remote address [akka.tcp://DeployTarget@65.52
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:30
full error messages man
gotta have em
JoeCoJabba
@joecojabba
Aug 14 2016 02:31
k... reconfiguring hocon
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:31
use the compose mode in gitter to paste them
JoeCoJabba
@joecojabba
Aug 14 2016 02:47
server
2016-08-14 01:41:35.6520 INFO start
2016-08-14 01:50:03.0736 INFO start
2016-08-14 02:00:22.4503 INFO start
2016-08-14 02:05:53.2159 INFO start
2016-08-14 02:08:09.1065 INFO start
2016-08-14 02:11:10.8878 INFO start
2016-08-14 02:17:52.4815 INFO start
2016-08-14 02:19:03.5753 INFO start
2016-08-14 02:34:42.4704 INFO start
2016-08-14 02:34:43.2986 DEBUG subscribing [akka://DeployTarget/system/log1-NLogLogger#168868776] to channel Akka.Event.Debug
2016-08-14 02:34:43.3142 DEBUG subscribing [akka://DeployTarget/system/log1-NLogLogger#168868776] to channel Akka.Event.Info
2016-08-14 02:34:43.3142 DEBUG subscribing [akka://DeployTarget/system/log1-NLogLogger#168868776] to channel Akka.Event.Warning
2016-08-14 02:34:43.3142 DEBUG subscribing [akka://DeployTarget/system/log1-NLogLogger#168868776] to channel Akka.Event.Error
2016-08-14 02:34:43.3142 DEBUG Logger log1-NLogLogger [NLogLogger] started
2016-08-14 02:34:43.3142 DEBUG subscribing [akka://DeployTarget/system/UnhandledMessageForwarder#1492509064] to channel Akka.Event.UnhandledMessage
2016-08-14 02:34:43.3142 DEBUG StandardOutLogger being removed
2016-08-14 02:34:43.3298 DEBUG unsubscribing [akka://all-systems/] from all channels
2016-08-14 02:34:43.3298 DEBUG now supervising akka://DeployTarget/system/UnhandledMessageForwarder
2016-08-14 02:34:43.3298 DEBUG Started (Akka.Event.LoggingBus+UnhandledMessageForwarder)
2016-08-14 02:34:43.3298 DEBUG Default Loggers started
2016-08-14 02:34:43.3767 INFO Starting remoting
2016-08-14 02:34:43.3767 DEBUG now supervising akka://DeployTarget/system/remoting-terminator
2016-08-14 02:34:43.4080 DEBUG now supervising akka://DeployTarget/system/transports
2016-08-14 02:34:43.4080 DEBUG Starting prune timer for endpoint manager...
2016-08-14 02:34:43.4080 DEBUG Started (Akka.Remote.EndpointManager)
2016-08-14 02:34:43.4080 DEBUG now supervising akka://DeployTarget/system/endpointManager
2016-08-14 02:34:43.4080 DEBUG Started (Akka.Remote.TransportSupervisor)
2016-08-14 02:34:43.4392 DEBUG Started (Akka.Remote.RemoteActorRefProvider+RemotingTerminator)
2016-08-14 02:34:43.4392 DEBUG now watched by [akka://DeployTarget/system]
2016-08-14 02:34:43.6267 DEBUG Started (Akka.Remote.Transport.AkkaProtocolManager)
2016-08-14 02:34:43.6267 DEBUG now supervising akka://DeployTarget/system/transports/akkaprotocolmanager.tcp.0
2016-08-14 02:34:43.6580 INFO Remoting started; listening on addresses : [akka.tcp://DeployTarget@65.52.244.82:8090]
2016-08-14 02:34:43.6580 INFO Remoting now listens on addresses: [akka.tcp://DeployTarget@65.52.244.82:8090]
2016-08-14 02:34:43.6580 DEBUG now supervising akka://DeployTarget/system/remote-watcher
2016-08-14 02:34:43.6892 DEBUG Started (Akka.Remote.RemoteWatcher)
2016-08-14 02:34:43.6892 WARN {
  akka : {
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    stdout-loglevel : DEBUG
    loglevel : DEBUG
    log-config-on-start : on
    actor : {
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
      provider : "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
      serializers : {
        wire : "Akka.Serialization.WireSerializer, Akka.Serialization.Wire"
      }
      serialization-bindings : {
        System.Object : wire
      }
    }
    remote : {
      helios : {
        tcp : {
          port : 8090
          hostname : 0.0.0.0
          public-hostname : 65.52.244.82
        }
      }
    }
  }
}
2016-08-14 02:34:43.6892 DEBUG Started (Akka.Remote.RemoteDeploymentWatcher)
2016-08-14 02:34:43.6892 DEBUG now supervising akka://DeployTarget/system/remote-deployment-watcher
2016-08-14 02:34:43.6892 DEBUG now supervising akka://DeployTarget/system/deadLetterListener
2016-08-14 02:34:43.6892 DEBUG now supervising akka://DeployTarget/system/EventStreamUnsubscriber-1
2016-08-14 02:34:43.6892 DEBUG subscribing [akka://DeployTarget/system/deadLetterListener#1832438538] to channel Akka.Event.DeadLetter
the information you need to debug this is whether or not messages are going through
those two configuration settings will help
JoeCoJabba
@joecojabba
Aug 14 2016 02:49
k thanks
Aaron Stannard
@Aaronontheweb
Aug 14 2016 02:49
both received and sent messages
JoeCoJabba
@joecojabba
Aug 14 2016 02:59
here is client
2016-08-13 20:57:39.7863 DEBUG subscribing [akka://Deployer/system/log1-NLogLogger#528655218] to channel Akka.Event.Debug
2016-08-13 20:57:39.8073 DEBUG subscribing [akka://Deployer/system/log1-NLogLogger#528655218] to channel Akka.Event.Info
2016-08-13 20:57:39.8073 DEBUG subscribing [akka://Deployer/system/log1-NLogLogger#528655218] to channel Akka.Event.Warning
2016-08-13 20:57:39.8073 DEBUG subscribing [akka://Deployer/system/log1-NLogLogger#528655218] to channel Akka.Event.Error
2016-08-13 20:57:39.8073 DEBUG Logger log1-NLogLogger [NLogLogger] started
2016-08-13 20:57:39.8073 DEBUG subscribing [akka://Deployer/system/UnhandledMessageForwarder#1206892840] to channel Akka.Event.UnhandledMessage
2016-08-13 20:57:39.8073 DEBUG now supervising akka://Deployer/system/UnhandledMessageForwarder
2016-08-13 20:57:39.8073 DEBUG Started (Akka.Event.LoggingBus+UnhandledMessageForwarder)
2016-08-13 20:57:39.8073 DEBUG StandardOutLogger being removed
2016-08-13 20:57:39.8073 DEBUG unsubscribing [akka://all-systems/] from all channels
2016-08-13 20:57:39.8073 DEBUG Default Loggers started
2016-08-13 20:57:39.8073 INFO Starting remoting
2016-08-13 20:57:39.8073 DEBUG now supervising akka://Deployer/system/remoting-terminator
2016-08-13 20:57:39.8073 DEBUG now supervising akka://Deployer/system/transports
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/endpointManager
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.TransportSupervisor)
2016-08-13 20:57:39.8163 DEBUG Starting prune timer for endpoint manager...
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.EndpointManager)
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.RemoteActorRefProvider+RemotingTerminator)
2016-08-13 20:57:39.8163 DEBUG now watched by [akka://Deployer/system]
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.Transport.AkkaProtocolManager)
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/transports/akkaprotocolmanager.tcp.0
2016-08-13 20:57:39.8163 INFO Remoting started; listening on addresses : [akka.tcp://Deployer@0.0.0.0:64947]
2016-08-13 20:57:39.8163 INFO Remoting now listens on addresses: [akka.tcp://Deployer@0.0.0.0:64947]
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/remote-watcher
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/remote-deployment-watcher
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/deadLetterListener
2016-08-13 20:57:39.8163 DEBUG subscribing [akka://Deployer/system/deadLetterListener#1537357914] to channel Akka.Event.DeadLetter
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Event.DeadLetterListener)
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/EventStreamUnsubscriber-1
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.RemoteDeploymentWatcher)
2016-08-13 20:57:39.8163 DEBUG registering unsubscriber with Akka.Event.EventStream
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Event.EventStreamUnsubscriber)
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.RemoteWatcher)
2016-08-13 20:57:39.8163 WARN {
  akka : {
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    stdout-loglevel : DEBUG
    loglevel : DEBUG
    log-config-on-start : on
    actor : {
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
      provider : "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
      serializers : {
        wire : "Akka.Serialization.WireSerializer, Akka.Serialization.Wire"
      }
      serialization-bindings : {
        System.Object : wire
      }
    }
    remote : {
      log-received-messages : on
      log-sent-messages : on
      helios : {
        tcp : {
          port : 0
          hostname : 0.0.0.0
        }
      }
    }
  }
}
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.ReliableDeliverySupervisor)
2016-08-13 20:57:39.8163 DEBUG now watched by [akka://Deployer/system/endpointManager#564522885]
2016-08-13 20:57:39.8163 DEBUG now supervising akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1/endpointWriter
2016-08-13 20:57:39.8163 DEBUG Started (Akka.Remote.EndpointWriter)
2016-08-13 20:57:39.8313 DEBUG now watched by [akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1#1746871028]
2016-08-13 20:57:39.8313 DEBUG now supervising akka://Deployer/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1
2016-08-13 20:57:39.8313 DEBUG Started (Akka.Remote.Transport.ProtocolStateActor)
2016-08-13 20:57:54.8948 WARN AssociationError [akka.tcp://Deployer@0.0.0.0:64947] -> akka.tcp://DeployTarget@65.52.244.82:8090: Error [Invalid address: akka.tcp://DeployTarget@65.52.244.82:8090] []
2016-08-13 20:57:54.9198 DEBUG Stopped
2016-08-13 20:57:54.9408 WARN Tried to associate with unreachable remote address [akka.tcp://DeployTarget@65.52.244.82:8090]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://DeployTarget@65.52.244.82:8090] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: failed as a result of System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectTimeoutException: connection timed out: 65.52.244.82:8090
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectTimeoutException: connection timed out: 65.52.244.82:8090
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectTimeoutException: connection timed out: 65.52.244.82:8090
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
---> (Inner Exception #0) Helios.Channels.ConnectTimeoutExceptio
apologies at my newb gitter/akka level
   --- End of inner exception stack trace ---]
2016-08-13 20:57:54.9408 ERROR Invalid address: akka.tcp://DeployTarget@65.52.244.82:8090
2016-08-13 20:57:54.9608 DEBUG no longer watched by [akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1#1746871028]
2016-08-13 20:57:54.9608 DEBUG Stopping
2016-08-13 20:57:54.9688 DEBUG Disassociated [akka.tcp://Deployer@0.0.0.0:64947] -> akka.tcp://DeployTarget@65.52.244.82:8090
2016-08-13 20:57:54.9688 DEBUG Stopped
2016-08-13 20:57:54.9688 INFO Message AckIdleCheckTimer from akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1/endpointWriter to akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1/endpointWriter was not delivered. 1 dead letters encountered.
2016-08-13 20:57:54.9688 DEBUG Stopped
2016-08-13 20:57:54.9688 DEBUG received AutoReceiveMessage <Terminated>: [akka://Deployer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDeployTarget%4065.52.244.82%3A8090-1#1746871028] - ExistenceConfirmed=True
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:04
oh man, I think I know the issue here
this is a IP protocol thing
using 0.0.0.0 for any TCP operations aside from accept is a no-no; you can listen on all incoming addresses there, but the system can't open an outbound client using that address
I looked quickly but couldn't find a source online for that which didn't diverge down the black hole of NETSEC despair
just try changing the bound IP of your client to 127.0.0.1
then that should work
it's a special address designed for being able to listen on all network interfaces
(0.0.0.0)
JoeCoJabba
@joecojabba
Aug 14 2016 03:06
hmm... k
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:06
can't be used for outbound connections
JoeCoJabba
@joecojabba
Aug 14 2016 03:07
server log was the same just had the extra log message settings
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:07
yeah, the issue is that the client can't open its outbound connection
it's a socket-level exception
Akka.Remote hasn't even had a chance to run yet
(in the context of this error)
JoeCoJabba
@joecojabba
Aug 14 2016 03:10
hmm... k... I will work that through and see where i get....
so to recap
is it fine for the server to set port, hostname=0.0.0.0 and public-hostname=public-ip
should it ever list the internal ip of the vm? eg its 10.x.x.x address?
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:13
if you're communicating 100% within your Azure Security Group
meaning everyone who's using Akka.Remote is on the same NAT
then using a private address is fine
public address is only needed when an external client or service needs to connect
JoeCoJabba
@joecojabba
Aug 14 2016 03:13
and for the client, port = 0, hostname=127.0.0.1 and then akka:// address pointing to <systemname>@public-hostname
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:14
in a networked scenario the client should be bound to an IP that is specific to the machine it's running on
so you might have to grab the list of IP addresses from the local machine and pick one
if you want to do that programmatically
JoeCoJabba
@joecojabba
Aug 14 2016 03:15
k thanks... I'll see where I land... running the client both external and both local to the server VM hasnt get me anywhere using anything but localhost.
appreciate the awesome support!
Aaron Stannard
@Aaronontheweb
Aug 14 2016 03:15
you're welcome!
sounds like this is just a socket addressing issue
go ahead and ask questions if have anymore
and if you can't get anyone here, we do check StackOverflow for akka.net tags
JoeCoJabba
@joecojabba
Aug 14 2016 03:16
:+1:
JoeCoJabba
@joecojabba
Aug 14 2016 03:36
@Aaronontheweb winning!
the combination was Server 
...
    port = 8090
    hostname = 0.0.0.0
    public-hostname=65.52.244.82
VM client
port = 0
hostname = local ip
public-hostname=65.x.x.x
JoeCoJabba
@joecojabba
Aug 14 2016 03:41
and the external client
port = 0
hostname= 0.0.0.0
the only oddity occuring now is that a very simple actor (Echo example) works with both clients, however my own actor only works when client/server are on the same machine.... the local->VM fails with a "subject is null" on the ActorSelection
JoeCoJabba
@joecojabba
Aug 14 2016 03:47
no my mistake... its all working as expected....
thanks again
now to recycle that VM and grab a new public IP!
Chris G. Stevens
@cgstevens
Aug 14 2016 21:10
I am trying to track this down but it seems like when I get this error the member because unreachable by other members and is removed from the cluster.
Running a cluster singleton the member that is remove is actually still running so will need to put in a check for that but I don't know what causes:
Error while decoding incoming Akka PDU
Aaron Stannard
@Aaronontheweb
Aug 14 2016 22:14
this is with Cluster Singleton?
mind posting an error message and creating a Github issue?