These are chat archives for akkadotnet/akka.net

9th
Nov 2015
Patrick Kelly
@featuresnap
Nov 09 2015 01:13
I was wondering what I need to do to have a RoundRobinPool ONLY restart the child which threw an exception. I have configured the pool using HOCON configuration, and would like to know if there is another HOCON setting I can apply to make the pool NOT escalate failures to its parent and restart the whole pool. Or do I have to configure the supervisor strategy for a pool in code?
Yin Zhang
@melcloud
Nov 09 2015 01:22
@featuresnap I don't believe you can do it in config. Code is your only option
Patrick Kelly
@featuresnap
Nov 09 2015 01:27

@melcloud Thanks. So if I have the following line that turns my actor into a pool based on config:

            //ConversionWorker
            Props workerProps = actorSystem.DI().Props<ConversionWorker>().WithRouter(FromConfig.Instance);

Is there a simple statement I can add on to the Props declaration above to add the appropriate supervisor strategy through code?

Yin Zhang
@melcloud
Nov 09 2015 01:28
            .WithRouter(FromConfig.Instance)
            .WithSupervisorStrategy(new OneForOneStrategy(new DeployableDecider(Directive.Restart))),
@featuresnap
Patrick Kelly
@featuresnap
Nov 09 2015 01:28
@melcloud nice. Thanks!
Yin Zhang
@melcloud
Nov 09 2015 01:28
@featuresnap :smile:
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 06:32
@melcloud problem lies in supervision strategy, which for some reason is not applied, once child throws an exception.
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 06:37
I've created an issue #1411 for your case
Arjen Smits
@Danthar
Nov 09 2015 08:27
@melcloud you need to give the backoffsupervisor a supervisionstrategy that returns a stopping directive when an error is received
made the same mistake. The backoffsupervisor does not use the supervisionstrategy of the child its monitoring.
Zetanova
@Zetanova
Nov 09 2015 08:27
@Horusiath What's is a good snapshot strategie? I thinking about something dynamic calculated with event count per time interval
Arjen Smits
@Danthar
Nov 09 2015 08:28
so when you dont give it a strategy, it defaults to a strategy that restarts
@Zetanova What i normally do is simply pick a number (for example 10) based on an educated guess about how the system is expected to behave in terms of traffic to an AR
then i monitor how long it takes for an AR to load.
if it starts to take to long, you can think about making more snapshots to reduce the IO load.
I did build a system which dynamically attempts to determine the count, but it ended badly :P
Zetanova
@Zetanova
Nov 09 2015 08:35
@Danthar i am unloading the AR after a timeout, maybe only to make on unload a snapshot with any number of events
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 08:36
@Zetanova time interval is good mostly for scenarios, when you have stable frequent stream of events persisted. But most of the time better option is to have snapshots stored per each X events persisted. You can use separate filed as persist counter if X is relatively low (so you don't need to persist it) or use LastSequenceNr and save snapshot on LastSequenceNr % X == 0
Zetanova
@Zetanova
Nov 09 2015 08:42
@Danthar @Horusiath thx for the advice, i will try with if ((eventsSinceLastSnapshot++) >= MaxEventsToSnapshot) on Persists and if (eventsSinceLastSnapshot >= MinEventsToSnapshot ) on PostStop
Zetanova
@Zetanova
Nov 09 2015 08:49
How and when to discard snapshots?
If an AR gets X snapshots a day, it should be fine to discard all beside the first snapshat of yesterday
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 08:57
depends on how you use them. You may use snapshots for sending your actors back into the future and showing their state at particular point in time. But if you need only the latest one, then on SaveSnapshotSuccess you could send next request to delete all snapshots up to current timestamp and snapshot number
SaveSnapshotSuccess contains metadata with timestamp and sequenceNr that can be used to by DeleteSnapshots.Criteria field to limit the range (remember only that values provided in criteria are inclusive, so you'd need to lower seqNr if you don't want to delete saved snapshot)
Zetanova
@Zetanova
Nov 09 2015 09:05
@Horusiath Yes, this back into the future will maybe be used. But then not all of the snapshots need to be there, to leave one shnapshot a day should be fine. With the DeleteSnapshots.Criteria its only possible to delete snapshots to a point in time, but not to delete a range
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 09:05
if(message is SaveSnapshotSuccess)
{
    var success = (SaveSnapshotSuccess)message;
    DeleteSnapshots(new SnapshotSelectionCriteria(success.Metadata.SequenceNr-1, success.Metadata.Timestamp));
}
Zetanova
@Zetanova
Nov 09 2015 09:05
yes and how to leave one each day?
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 09:06
select appropriate timestamp value ;) it's DateTime
Zetanova
@Zetanova
Nov 09 2015 09:06
its a max
it would delete all snapshots before
if the AR has 100 snapshots a day
would be good to leave only the first and delete the rest
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 09:08
ok, for that you'll need to work on journal backend itself (in case of SQL, execute command with appropriate query)
Zetanova
@Zetanova
Nov 09 2015 09:10
is it possible with engines like EventStore?
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 09:17
this is something specific for a particular datastore backend. Akka.Persistence doesn't have this option by default, but you can inherit from a particular journal and override it's behavior to handle additional messages
so if you use EventStore, then you need to figure out how to do this in eventstore
AFAIK persistence id, seqNr and timestamp are part of the standard schema for all snapshots, so any persistence provider should store them in some way
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 09:25
buuu, on 8 cores I had 30mln msg/sec - it's "only" 4x improvement on 6x cores ;P
Roger Johansson
@rogeralsing
Nov 09 2015 09:26
ye, we need linear scalablility :P
Zetanova
@Zetanova
Nov 09 2015 09:34
dont forget NUMA
Yin Zhang
@melcloud
Nov 09 2015 10:05
@Horusiath @Danthar Thanks guys. Yeah, that supervisor strategy is the problem.
Thomas Lazar
@thomaslazar
Nov 09 2015 10:47
@Aaronontheweb @rogeralsing i have been to a NHibernate course given by Oren Eini (http://ayende.com/blog/ he worked on NHibernate quite a bit a while back) and iirc he said you shouldn't access an ISession (not the ISessionFactory) from another thread. but that was a couple of years ago. could just remember it wrong.
Arjen Smits
@Danthar
Nov 09 2015 10:56
@thomaslazar I used to work with NHibernate as well. And I remember that as well. I believe it had something todo with some internal state being thread-static. But that was a long while back. Not sure if it still applies.
Thomas Lazar
@thomaslazar
Nov 09 2015 10:59
@Danthar i guess i'll just ask on the nhusers google groups what the current state is. i will share my findings here then
Yin Zhang
@melcloud
Nov 09 2015 11:08
Hi guys, if I start an Akka system in web API, how can I make sure IIS won't kill the actor system when it is idle?
Thomas Lazar
@thomaslazar
Nov 09 2015 11:09
you can't
from what i heard here and in other places it's best to move the actual akka systems stuff doing work into it's own service and have the IIS actor system communicate to it through Akka.Remote.
so if IIS kills of the actor system it's just the message passing one
Yin Zhang
@melcloud
Nov 09 2015 11:12
@thomaslazar Mmm... I do have my main logic in a windows service, however, I still need API to delivery them through remote. Is there anyway to restart the actor system without recycle or shutdown the application?
Thomas Lazar
@thomaslazar
Nov 09 2015 11:16
that i don't know. maybe Akka.Persistence could help you save your state on shutdown. but i haven't used that stuff yet. i just know you can't keep IIS from killing off your application on a whim.
Yin Zhang
@melcloud
Nov 09 2015 11:18
@thomaslazar I got persistence to save state. It is more off when it is shutdown, you need another request to make it alive. slightly annoying ... I probably do a ping health check then. Thanks!
Thomas Lazar
@thomaslazar
Nov 09 2015 11:19
ping health check will not keep IIS from killing off the app after a certain amount of time. it will die eventually. at least that's what mine does, and i do ping them regularly.
Yin Zhang
@melcloud
Nov 09 2015 11:21
@thomaslazar Mmm... ok. Let me set the idle timeout to zero for now. Probably should convert this to host in windows service soon
John Nicholas
@MrTortoise
Nov 09 2015 12:26
Im having an annoyance with logging in unit tests ... how cna i flush the logs to get all the output before the next test? I have tried shutting down the actor system between tests but when i restart i still get log messages from a previous test.
the question is equally how when an actor system fails - and i am control of how it crumples - can i ensure that all logged messages have been outputted.
Chris G. Stevens
@cgstevens
Nov 09 2015 14:28

I am still tracking down why, when I connect from my laptop to my dev environment I have a 90% chance of bringing down my cluster… All members start throwing an association error which even Lighthouse starts to log System.OutOfMemoryException. Issue starts when 1.9.2.8:49514 becomes unreachable by any cluster member… for over an hour each member logged about 600 events for the first error and all 5 members started going from the first error to the second error. Anyone else run into this?

Logger: Akka.Actor.OneForOneStrategy
Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://MyService@1.9.2.8:49514 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: No connection could be made because the target machine actively refused it 1.9.2.8:49514 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 1.9.2.8:49514
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__6.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)

Logger: Akka.Actor.OneForOneStrategy
Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://MyService@1.9.2.8:49514 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start()
at Helios.Concurrency.DedicatedThreadPool.PoolWorker..ctor(WorkerQueue work, DedicatedThreadPool pool, Boolean errorRecovery)
at Helios.Concurrency.DedicatedThreadPool..ctor(DedicatedThreadPoolSettings settings)
at Helios.Concurrency.Impl.DedicatedThreadPoolFiber..ctor(IExecutor executor, Int32 numThreads)
at Helios.Net.Bootstrap.ClientBootstrap.get_EventLoop()
at Helios.Net.Bootstrap.TcpConnectionFactory.CreateConnection(INode localEndpoint, INode remoteEndpoint)
at Helios.Net.Bootstrap.ClientConnectionFactoryBase.NewConnection(INode localEndpoint, INode remoteEndpoint)
at Akka.Remote.Transport.Helios.HeliosTransport.NewClient(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__6.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)

John Nicholas
@MrTortoise
Nov 09 2015 14:48
@cgstevens is this a different out of memory exception to what you were getting the other day?
Chris G. Stevens
@cgstevens
Nov 09 2015 16:19
@MrTortoise No it is the same... I thought it was something I did... but when it is happening on all of my members including Lighthouse I am wondering if there is something else going on within the cluster.
Aaron Stannard
@Aaronontheweb
Nov 09 2015 17:11
@rogeralsing @Horusiath I demand an async / await benchmark ducks
seriously though, that's awesome - you won't get 100% linear scalability due to scheduling overhead but still, that's pretty amazing
micro-optimization: processor-pinned dispatcher :p
is that even possible to do in managed code? I have no idea
mggillen
@mggillen
Nov 09 2015 17:44
I have a router, a Worker Controller, and some worker actors. What is the correct way to issue a stop-shutdown to the router once, say, all the messages in a workload have been received?
var system = ActorSystem.Create("MySystem");
var controller = system.ActorOf(Props.Create(() => new Actors.JobController(aRowEnumeration.Count())),"JobController");
var router = system.ActorOf(Props.Create(() => new Actors.WorkerActor(controller)).WithRouter(new RoundRobinPool(5)), "MyRouter");
mggillen
@mggillen
Nov 09 2015 17:54
The issue I get is that if I use the Context.Stop(Self) where Self is the controller actor the system hangs when I reach the System.AwaitTermination which tells me (no pun intended) that I'm not shutting down the router.
Zetanova
@Zetanova
Nov 09 2015 17:56
Does somebody have a good Exclusive SynchronizationContext ? I need it for Powershell WirteObject and mine sometimes just deathlocks. It's like that the ContinueWith(() => syncCtx.Complete()) is never called. Cant find nothing related in the Inet. I am thinking it has something to to with the "work-stealing algorithm" of the TaskPool
Natan Vivo
@nvivo
Nov 09 2015 18:02
hey @Aaronontheweb , just bought the "Writing High Performance Code in .NET". Thanks for the tip (has been a while)
@melcloud you can work around IIS shuttind down the app if you really want. it's still better to move code to a service, though. Take a look at the IRegisteredObject here: http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx/ - it explains how to register an object so that it's called when IIS decides to shutdown
Natan Vivo
@nvivo
Nov 09 2015 18:10
This message was deleted
Aaron Stannard
@Aaronontheweb
Nov 09 2015 19:34
@nvivo awesome Natan! That book is amazing
I learned so many low-level details of the CLR and type system from that
it's right up there with CLR via C#
Christian Duhard
@cduhard
Nov 09 2015 19:39
what should I use to deserialze Akka.Persistence events outside of a PersistentActor?
Natan Vivo
@nvivo
Nov 09 2015 19:43
@Aaronontheweb I read a book few years back that is also very good to understand some internals: http://www.amazon.com/Inside-Microsoft-Net-Il-Assembler/dp/0735615470
it's quite old but almost everything it talks about is still used today.
Christian Duhard
@cduhard
Nov 09 2015 19:46
i'm trying to consume a projection of persistentactor events via eventstore
Zetanova
@Zetanova
Nov 09 2015 19:50
@cduhard Currently with PersistentView
Christian Duhard
@cduhard
Nov 09 2015 19:51
what i want is essentially this:
return (IPersistentRepresentation) serializer.FromBinary(@event.Data, typeof (IPersistentRepresentation));
i am using Competing Consumers in EventStore
but i need to deserizle the akka persistence events
Christian Duhard
@cduhard
Nov 09 2015 19:58
I am consuming projections, would PersistentView be relevant here?
Zetanova
@Zetanova
Nov 09 2015 20:18
@cduhard sry dont realy know. If u need to trasnform/process the Data/Events of a single AR in a different Form then u can go for PersistentView else u can Subscribe to EventBus. If u need to work with multiple Events of different AR's u neet to query the Journal Actor directly.
Christian Duhard
@cduhard
Nov 09 2015 20:30
I am reading the events outside of any actorsystem
Zetanova
@Zetanova
Nov 09 2015 20:49
This message was deleted
This message was deleted
This message was deleted
On the IPersistentRepresentation is a Payload and Manifest Property
with it u can deserialize the Payload
Christian Duhard
@cduhard
Nov 09 2015 21:10
yeah, i'm trying, unsuccessfully
Zetanova
@Zetanova
Nov 09 2015 21:22
private object GetPayload(DbDataReader reader, string manifest)
        {
            var type = Type.GetType(manifest, true);
            var binary = (byte[]) reader[PayloadIndex];

            var serializer = _serialization.FindSerializerForType(type);
            return serializer.FromBinary(binary, type);
        }
Its of the DefaultJournalQueryMapper
Bartosz Sypytkowski
@Horusiath
Nov 09 2015 21:33
@cduhard you can customize the way you're serializing event's - by default payload field is UTF8 binary with JSON inside
Christian Duhard
@cduhard
Nov 09 2015 21:34
i've got it working
            var representation = _deserializer.GetRepresentation(_serializer, @event.Event);
            var payLoad = representation.Payload;
@Zetanova i'm not using a Sql based journal
still getting those wacky Debug messages on replay
the Sender of a dead pool actor is getting serialized with the persisted event
Zetanova
@Zetanova
Nov 09 2015 21:38
How to get a task schudler or synchronization context in the current actor message process ? Need Something that loads the ActorContext back
It worked good with ActorTaskSchulder.Run() and RX.Subcription inside
but sometimes the processing stopped or maybe the message processing of the actor
Aaron Stannard
@Aaronontheweb
Nov 09 2015 21:42
@cgstevens what are your cluster settings?
i.e. is auto-down turned on and what are the logging settings?
Zetanova
@Zetanova
Nov 09 2015 21:47
var subscription = exec
                .ObserveOn(CurrentThreadScheduler.Instance)
                .Subscribe(
                    onNext: n =>
                    {
                        Sender.Tell(n);
                    },
                    onError: ex =>
                    {
                        Sender.Tell(new PNetEvents.Failed(ex, PNetDate.UtcNowOffset));
                        Context.Parent.Tell(Passivate.Instance);
                    },
                    onCompleted: () =>
                    {
                        Sender.Tell(new PNetEvents.Completed(PNetDate.UtcNowOffset));
                        Context.Parent.Tell(Passivate.Instance);
                    }
                );
I would need "Just" a Schudler or SyncContext that will recreate the ActorContext and i would have the same behaivor as PipeTo
Is there something?
Zetanova
@Zetanova
Nov 09 2015 23:56
I created a RX Schudler that pushes CompleteTask system-messages to the Akka Schudler, works perfectly