These are chat archives for akkadotnet/akka.net

19th
Mar 2018
Havret
@Havret
Mar 19 2018 09:25 UTC
Are there any special tooling to test PersistentFSM?
troasfl
@troasfl
Mar 19 2018 15:51 UTC
hi, please is there a way of registering ApplicationDbContext so that it can be injected into an Actor in .net core 2.0?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 15:52 UTC
@troasfl if we don't have a DI plugin that supports that, just pass in the DI container itself as a constructor argument
should be noted though: passing in a short-lived object like a database connection into an actor's constructor is a total anti-pattern in actor-based programming
actors can live forever theoretically
database connections are intended to be short-lived objects
therefore, actors should be armed with a tool to be able to re-acquire a short-lived dependency at its leisure
troasfl
@troasfl
Mar 19 2018 15:54 UTC
ok
Aaron Stannard
@Aaronontheweb
Mar 19 2018 15:54 UTC
passing in the DI Container itself is one way of doing that
reason I mention this is because the only major downtime I've had with an Akka.NET app in production was as a result of making that mistake myself
passed in a CassandraContext into the constructor of my actors that wrote to the database
worked great for the first 60 seconds until the Cassandra driver pooled all of its outstanding contexts
and disposed them from out underneath the actor
troasfl
@troasfl
Mar 19 2018 15:56 UTC
interesting
Aaron Stannard
@Aaronontheweb
Mar 19 2018 15:56 UTC
yeah
troasfl
@troasfl
Mar 19 2018 15:56 UTC
So whats the recommended way in .netcore
Aaron Stannard
@Aaronontheweb
Mar 19 2018 15:56 UTC
it's just a difference between Akka.NET and something like ASP.NET
where the controllers you inject into live for only 1 request
so they're even more short-lived than a database connection is
I use a technique called functional props
where I pass in the following syntax into my database actor's constructor
(grabbing some source rq)
public class IdentityStoreActor : ReceiveActor
    {
        #region Message classes

        public class CheckForAvailableUserName
        {
            public CheckForAvailableUserName(string displayName)
            {
                DisplayName = displayName;
            }

            public string DisplayName { get; private set; }
        }

        public class UserNameAvailablity
        {
            public UserNameAvailablity(string displayName, bool isAvailable)
            {
                IsAvailable = isAvailable;
                DisplayName = displayName;
            }

            public string DisplayName { get; private set; }

            public bool IsAvailable { get; private set; }
        }

        public class FetchAuthenticationInformation
        {
            public FetchAuthenticationInformation(string displayName)
            {
                DisplayName = displayName;
            }

            public string DisplayName { get; private set; }
        }

        public class FetchUserByName
        {
            public FetchUserByName(string displayName)
            {
                DisplayName = displayName;
            }

            public string DisplayName { get; private set; }
        }

        public class FetchUserByCookie
        {
            public FetchUserByCookie(Guid cookieId)
            {
                CookieId = cookieId;
            }

            public Guid CookieId { get; private set; }
        }

        #endregion

        private readonly Func<IUserRepository> _userRepoFactory;

        public IdentityStoreActor(Func<IUserRepository> userRepoFactory)
        {
            _userRepoFactory = userRepoFactory;

            Receive<CheckForAvailableUserName>(name =>
            {
                var repo = _userRepoFactory();
                var sender = Sender;
                var result = repo.UserNameIsAvailable(name.DisplayName).ContinueWith(tr =>
                {
                    repo.Dispose();
                    if (tr.Result.Payload == true)
                    {
                        return new UserNameAvailablity(name.DisplayName, true);
                    }
                    return new UserNameAvailablity(name.DisplayName, false);
                }).PipeTo(sender);
            });
truncated some of the source here
but this is the key portion
public IdentityStoreActor(Func<IUserRepository> userRepoFactory)
I just pass in a delegate I can use to get an instance of my repository or db context or whatever
in production, here's what this does:
private static IKernel SetupNinject()
        {
            var kernel = new StandardKernel(new INinjectModule[] {new FactoryModule()});

            //SQL Schema information
            var connectionString = ConfigurationManager.ConnectionStrings["akkaChat"];
            var schemaName = ConfigurationManager.AppSettings["SqlSchemaName"];
            var chatTableName = ConfigurationManager.AppSettings["ChatTableName"];
            var userTableName = ConfigurationManager.AppSettings["UsersTableName"];

            //Reuse the same instance
            kernel.Bind<SqlQueries>()
                .ToMethod(context1 => new SqlQueries(schemaName, userTableName, chatTableName))
                .InSingletonScope();

            kernel.Bind<IDbConnection>()
                .To<SQLiteConnection>()
                .WithConstructorArgument("connectionString", connectionString.ConnectionString);

            kernel.Bind<IUserRepository>().To<SqlUserRepository>();
            kernel.Bind<IRoomRepository>().To<SqlRoomRepository>();

            kernel.Bind<Func<IUserRepository>>().ToMethod(context1 => () => context1.Kernel.Get<IUserRepository>());
            kernel.Bind<Func<IRoomRepository>>().ToMethod(context1 => () => context1.Kernel.Get<IRoomRepository>());

            //Cryptography information
            kernel.Bind<ICryptoService>()
                .To<HMACSHA1Service>()
                .WithConstructorArgument("validationKey", ConfigurationManager.AppSettings["ValidationKey"]);

            return kernel;
        }
kernel.Bind<Func<IUserRepository>>().ToMethod(context1 => () => context1.Kernel.Get<IUserRepository>()); - actually uses Ninject internally
Aaron Stannard
@Aaronontheweb
Mar 19 2018 16:01 UTC
during dev / test I can just pass in a factory method instead without mocking a DI container
but more importantly, this gives the actor a tool to create its own connection instances at its leisure
rather than trying to do that from outside the actor with constructor-based DI
you'll get better results this way long-term IMHO
troasfl
@troasfl
Mar 19 2018 16:03 UTC
ok
great
thanks
Andrey Leskov
@andreyleskov
Mar 19 2018 16:09 UTC
Hi all, is there a way to define a supervision strategy for a shard?
Ondrej Pialek
@ondrejpialek
Mar 19 2018 16:11 UTC

Same situation?

but while we're in development we're still working on that

This is what I do:

truncate table EventJournal
truncate table MetaData
truncate table SnapshotStore
:)
Andrey Leskov
@andreyleskov
Mar 19 2018 16:12 UTC
I mean same intention from my side - want sharded entities restarted only on some specific errors
and stopped on others
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:09 UTC
So I am wondering, is it safe to use Sender.Tell to reply to sharded entities? Would it be a problem if they sent a message expecting a reply at some point in the future and before a reply is received they'd get re-balanced onto a different node?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:10 UTC
same deal as any remote actor
or really, any actor you don't have direct control over in that context
Sender.Tell is usually safe
but if you need to have a long-running set of interactions with that actor
always Context.Watch it
you'll get a Terminated message back if that actor is re-balanced
there's a very small chance that during the tiny window where you process the message you got back from the sharded entity
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:11 UTC
well, I still wouldn't know what to do then, I'd need to get hold of the shard proxy and send the reply that way
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:12 UTC
that it could be killed / rebalanced before thn
in which case you'd want to just send it back to the ShardProxy
I think of it like this\
what is the cost of one message going missing in this case?
if it only happens once in every 10 million messages
and the impact of it going missing isn't an irrecoverable failure
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:13 UTC
yeah cool - except there is probably no way to "fake that", like sending the sharded entitie's parent as a sender, because that is not the proxy, right?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:13 UTC
correct
it's not the parent
although technically you could do something like this
actually, crap
can't really do that
was going to pass in a reference to the shard region
into the entity actor
and have it fake that
but chicken and egg
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:14 UTC
yeah, if the node goes down same problem
hm
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:14 UTC
at some point in time
if you're trying your damndest to cover all the fault tolerance scenarios
you're going to end up having to subscribe to the cluster events
and get notified when nodes go down / etc
or just build in at least once delivery into any critical interactions
or have a sync-ing mechanism like push-and-pull you can use to recover from failures
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:15 UTC
unfortunately even at least once is suspicable to this right?
sharded entity sends msg, expecting a confirmation, but it might get rebalanced
but it won't woke up (ddata)
so it doesn't resend the message
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:16 UTC
ah I see\
no remember entities
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:16 UTC
but I think I will have to bite the bullet
and have a manager that wakes them up
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:16 UTC
with at least once delivery, I always split up the workload for reliable delivery
one actor that does the actual ACK / timeout / retry accounting around messages
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:16 UTC
aha, how?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:17 UTC
your AtLeastOnceDelivery actor implementation is the class responsible for this
I always pair it with a second actor who always runs inside the exact same process as that actor
the second actor is responsible for routing the message to the correct destination
because the actor you were trying to deliver the message to before might move
or be recreated in a new part of the network
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:18 UTC
hm
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:18 UTC
so this actor might be one that just forwards all messages to a shard proxy
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:18 UTC
that is an interesting thought
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:18 UTC
in most of my apps it's usually just a clustered router
so you split the responsibilities of "which messages are still not yet acknowledged" from "who can acknowledge it?"
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:21 UTC
yeah, I have a saga that sends messages to ARs, those are sharded entities, and that is sent via a proxy so that is fine. but sagas are also sharded entities so that ack might get lost, but yeah, a) they probably don't have to be sharded entities, and b) I could fake the sender as some cluster singleton that will forward the replies to their shard proxy... interesting
will give it some more thoughts
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:21 UTC
:+1:
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:21 UTC
thanks @Aaronontheweb
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:21 UTC
I try!
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:27 UTC
Actually I simply can't trust IActorRef if it is remote (for critical things anyway). So either I make sure my entities get resurrected to resend the message (a) or I need to somehow extract information from the message that will help me to figure out where to send the reply to on the node of the recipient (b) (and this local actor would figure out where to send the message further). I think a) is much simpler in this case
Final question - does Context.Watch work over the network?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:27 UTC
yep
it does
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:27 UTC
awesome
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:27 UTC
works differently in Akka.Remote and Akka.Cluster though
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:27 UTC
that helps :)
how come?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:28 UTC
in Akka.Cluster, Terminated only fires when the node leaves the cluster or gets downed
this is because Akka.Cluster is more sophisticated in terms of network partition awareness and tolerance than Akka.Remote
Akka.Cluster assumes network partitions are temporary
and automatically tries to reconnect
Akka.Remote is dumb
it does not assume that
it assumes that if I can't connect to the node
then that actor must be dead and the process must be offline
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:29 UTC
aha, that is ok, I just need to find out later about that
and ressurect, so a delay is fine
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:29 UTC
yep
you should be fine then
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:29 UTC
actually preferred if it reconnects
Aaron Stannard
@Aaronontheweb
Mar 19 2018 19:29 UTC
if a node reboots
Akka.Cluster will automatically down the old instance of the node's actors
and everyone gets a Terminated message for that
since it can detect restarts
Ondrej Pialek
@ondrejpialek
Mar 19 2018 19:30 UTC
yeah, the whole node lifecycle I am still not sure about, we are running in Azure Service Fabric with a lot of custom code to have it running as a stateless service, it does it's own thing when things go bleak, so will see how it behaves :)
but this is a good start, should be fine I think
thanks a lot Aaron!
Joshua Benjamin
@annymsMthd
Mar 19 2018 23:06 UTC
Is it possible for sharded cluster nodes to get into a forward message loop?

For instance on one node I see:
Forwarding request for shard [43] to [akka.tcp://ClusterSystem@10.233.69.239:2551/system/sharding/VehiclePositionActor#1483370170]
and other
Forwarding request for shard [84] to [akka.tcp://ClusterSystem@10.233.79.102:2551/system/sharding/VehiclePositionActor#2080696550]

now these nodes have a singleton queue listener that is generating these messages so it looks like they are hashing the message differently and getting stuck in a loop

This is being hosted in k8s and these nodes are being deleted and recreated whenever we push out changes. This is when we tend to see this popup
Joshua Benjamin
@annymsMthd
Mar 19 2018 23:21 UTC
akkadotnet/akka.net#3362 perhaps?
Aaron Stannard
@Aaronontheweb
Mar 19 2018 23:53 UTC
yeah could be
we're going to be releasing 1.3.6 with some patches for this this week
I'm in the process of tracking down yet another sharding-related bug possibly right now
but my reproduction experiment is a tad expensive :p