These are chat archives for akkadotnet/akka.net

8th
Jun 2015
Thomas Lazar
@thomaslazar
Jun 08 2015 08:31
@Aaronontheweb big props for that remote deploy video you posted! but Y U NO 1080p bro? :P
Roman Golenok
@shersh
Jun 08 2015 10:10
Arjen Smits
@Danthar
Jun 08 2015 10:13
Nope. But would love to.
Ben Gale
@BenGale
Jun 08 2015 10:16
I’m planning to very soon
Roman Golenok
@shersh
Jun 08 2015 10:23
@BenGale , like plugin? or hardcode telemetry.SendEvent?
Anyway, share your expierence plz. I'm thinking about using it in the near future
Roger Johansson
@rogeralsing
Jun 08 2015 10:39
Id like to know too
Ben Gale
@BenGale
Jun 08 2015 10:45
Honnestly I’m not sure, we’ve got it in use for some websites and we’d like to unify our metrics into one spot. At the moment we’re playing arround with the monitoring library to see what we can do. As it comes together I’ll blog it and post a link
We’re still feeling our way around to find the best way to use all of this awesome stuff
jcwrequests
@jcwrequests
Jun 08 2015 12:44
@rogeralsing I have been rethinking my approach as it pertains to your example. Since a message off a queue is essentially just a string, an XML string in my case, then it probably makes sense to have an Actor that manages pulling the message, another for de-serializing the message and finally another that does the actually processing. Within each message there is a natural key value to route through the consistent hash pool. So does it make sense to have hash pools for each level under that actor or should that actor manage it's children explicitly or am I just overthinking this? BTW I have found away around my previous issue. Thanks.
Roger Johansson
@rogeralsing
Jun 08 2015 12:48
without knowing any real context here, it does sound complex to do hash routing for each level. I would probably try keep that at a single hashrouter, just for debugging/understanding the code. regarding a separate actor for deserialization, I'd do that if there is a risk for failure here, e.g. schema changes or such.. otherwise I would most likely do that together with the actual pulling from the queue. .. but again, I have very little context on your specific problem
jcwrequests
@jcwrequests
Jun 08 2015 12:49
I am taking messages off a queue that are from a webhook that provides subscription information. The message format is XML
The messages are received the order they are created.
It's important that I maintain that order. That was my originally concerns but I have found a way to deal with that.
Now I am just working through does it make sense to create a hierarchy for processing the message because in some cases I may want to just ignore it completely and in others I actually want to process it.
I don't know what I was thinking with the multi level hash thing. Still waking up.
Thanks @rogeralsing I think I am going to go with a top level actor to guard against schema changes.
jcwrequests
@jcwrequests
Jun 08 2015 12:54
And grab a cup of tea to wake up my brain.
:)
jnkramer3
@jnkramer3
Jun 08 2015 16:00

I am hoping for some help and understanding.
I am just starting to a development effort with Akka.net and clusters.
While I have not used clusters or Akka before, I have developed distributed systems.
I have found that logs are one of the most important piece of information to help in diagnosing when a system mis-behaves. This is especially true for systems at a customer's site or internally if it's not being monitor all the time. With this in mind, I wanted to consolidate logs. I can't see how to configure the system to consolidate logs into 1 (or a few places). My initial thoughts would be to have all seed nodes to have a full copy of the logs. This would be the permute place for logs, but I could see other actors wanting to also monitoring all logs for analysis and/or GUI type of needs.

My question is has anybody done this, what are the best practices for logging in a cluster and do I need to develop my own logging actors to get this kind of behavior?

Also as a side question, what are the best practices for handling clock differences between nodes in a cluster. This is especially true for logs to help in knowing what has happened when compared to other nodes.

Alan Schrank
@alanschrank
Jun 08 2015 16:06
Hello, 1st time here, and very new to akka.net. I have a akka.net experiment that is throwing out of memory exceptions. I think it is because messages are being generated faster than they can be processed. for a list of directories I tell a actor in a pooled router "get directory size". That actor enumerates the files in the directory send a message to a output actor that writes the info to a database. It the enumerates the directories in the parent directory and for each does a Self.Tell "get directory size". Two questions: 1) when I tell self on this actor am I telling the router to distribute the message amongst the pool or am I telling the actual same actor? I think I am wrong to be telling self and I should tell Sender. 2) What happens when the mailbox gets too big? Is there an implementation that doesn't keep it all in memory? If so how do I use it. Thanks for any insight.
Ben Gale
@BenGale
Jun 08 2015 16:23
:point_up: June 8 2015 11:23 AM I started on an extenstion to the akka monitoring project this afternoon for application insights, I hope to finish it off tomorrow and I’ll post info in here.
Chris Martin
@trbngr
Jun 08 2015 16:29
@jnkramer3 re: logs I would use something like Azure Service Bus Event Hubs and simply create a consumer for them. You can even use Stream Analytics
Aaron Stannard
@Aaronontheweb
Jun 08 2015 16:44
awesome @BenGale - let me know when you publish it!
@thomaslazar I could do my native resolution in 1920x1080, but Camtasia caps my YouTube uploads at 720p :p
jnkramer3
@jnkramer3
Jun 08 2015 16:46
Thanks, I don't have any experience with Azure. So, I did not know about them. I am not sure where this project is going (it's for myself) and was not sure I azure was going to be used but it is always a possibility.
Aaron Stannard
@Aaronontheweb
Jun 08 2015 16:47
@jnkramer3 for logging, you can plug in whatever logging layer you want in Akka.NET. When I'm dealing with high-volume logs I usually do something like @trbngr suggested or perhaps pump those logs into a service like Splunk
a Splunk adapter for Akka.NET logging would actually be a good idea for a community NuGet package, now that I think about it :p
jnkramer3
@jnkramer3
Jun 08 2015 16:49
I never thought about a totally separate logging solution. Food for thought.
My big issue with logs was to get all of them in one place.
I don't think I am going to have high volume of logs.
Aaron Stannard
@Aaronontheweb
Jun 08 2015 16:51
in a distributed environment where you have a small number of logs, you should write to something like Azure Table Storage or SQL Server then
we support NLog with the Akka.Logging.NLog package, and if I recall correctly NLog has a SQL Server target
it might also have an Azure Table Storage target too but I don't know offhand
jnkramer3
@jnkramer3
Jun 08 2015 16:52
I will need to give that some thought. Again I was not thinking of a separate logging infrastructure.
Aaron Stannard
@Aaronontheweb
Jun 08 2015 16:58
@alanschrank so this is a fun problem that's easy to recreate when you have a highly concurrent framework like Akka.NET
and you've correctly pointed out the cause of the problem: the producer can produce faster than the consumer can consume
here's the cool part: if you can solve this problem for a trivial / naive use-case inside a single app
then you can solve it for big, production-ready use cases too!
so when I've run into this problem before, the way I've solved it is with some implementation of the Throttle pattern: http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2
that's Akka JVM documentation I've justed linked to
but all of those concepts are 1:1 applicable and translatable to Akka.NET and C#
one irony of the throttle pattern is that by reducing the rate at which you produce messages, you can actually increase throughput of the system
works by reducing contention and resource management on the consumer side of the equation
Aaron Stannard
@Aaronontheweb
Jun 08 2015 17:04
(maybe irony is the wrong word - "counter-intuitive" might be a better way to describe it)
let me know if that makes sense / if you need some help implementing it in C#!
Aaron Stannard
@Aaronontheweb
Jun 08 2015 17:11
@Horusiath so I have an Akka.Persistence question for you from one of our trainees - how do PersistentActor and AtLeastOnceDeliveryActor behave if their underlying database becomes unavailable?
offhand I don't know myself :p
Alan Schrank
@alanschrank
Jun 08 2015 17:39
@Aaronontheweb Makes sense in theory, I haven't reviewed link yet, but will, I've placed a sleep in the producer as bad as that may, be as a test.
Aaron Stannard
@Aaronontheweb
Jun 08 2015 17:42
@alanschrank well sure, thats just about the easiest type of throttle you can implement :p - not a bad way to kick the tires on the concept
I usually implement a batching system - where the producer is configured to only have X number of outstanding messages in-flight at once
and the consumer has to reply back with some acknowledgement before the producer can release another message
Alan Schrank
@alanschrank
Jun 08 2015 17:49
@Aaronontheweb How can I tell how big my mailbox is and, since I am recursing, will Self.Tell send my message to my owning router or myself?
Aaron Stannard
@Aaronontheweb
Jun 08 2015 17:50
Self.Tell will send the message to yourself
gauging the depth of the mailbox isn't something you can do via the public API. It's supposed to be invisible to you, even though we all know it's there.
better to implement that sort of behavior via a messaging contract between actors
"I send you X, you acknowledge X with Y, and then I send you more X"
Alan Schrank
@alanschrank
Jun 08 2015 17:56
@Aaronontheweb Thanks for your help. I get out of your hair after one follow-up. How do I send a message to my router (from the routee) so one of my peers can deal with it? Do I use Context to look up the address of the actor I created with .WithRouter or is there a way off of Self or Sender or some other object the actor knows?
Aaron Stannard
@Aaronontheweb
Jun 08 2015 17:59
Context.Parent
that will allow the routee to talk directly to its parent
the router, in this case
although there's a chance the router could route the message right back to the actor who wanted to forward it to a peer :p
the other thing you can do is send a GetRoutees message to the router
you'll get a Routees message back
for a Pool router, you'll get a list of ActorRefRoutees
so you can filter the Routees list for any routee that isn't equal to yourself
and send them a message that way
Bartosz Sypytkowski
@Horusiath
Jun 08 2015 20:56

how do PersistentActor and AtLeastOnceDeliveryActor behave if their underlying database becomes unavailable?

@Aaronontheweb if this problem occurs, Journal will send a message about failure back to the persistent actor:

  • for exception thrown during writing new events to Journal, it will send WriteMessagesFailed(exception) and then PersistenceFailure with details for each event, that could not be stored
  • for exception thrown during replaying events from the Journal, message will be RecoveryFailure(exception) (you can also override OnReplayFailure method which unwraps message retrieval)
  • for exception thrown during reading sequence number from the Journal, error message is ReadHighestSequenceNrFailure(exception)
Bartosz Sypytkowski
@Horusiath
Jun 08 2015 21:06
You can decide, how to react on those, but from the persistent actor perspective, usual recovery handlers and persist callbacks won't be invoked - so there is no risk of corrupting actor's state.
Aaron Stannard
@Aaronontheweb
Jun 08 2015 21:22
thanks @Horusiath - that's super helpful!