These are chat archives for akkadotnet/akka.net

15th
Feb 2016
joonhwan
@joonhwan
Feb 15 2016 02:35
Is there any way to make akka.net remote not dissassoicated while debugging? (or, is there any kinda timeout setting out there?)
Yin Zhang
@melcloud
Feb 15 2016 05:46
@joonhwan yes. Under remote, there is transport failure settings. Which you can change the heartbeat interval to a larger value.
@joonhwan
watch-failure-detector {
threshold = 100
acceptable-heartbeat-pause = 20 s
}
transport-failure-detector {
heartbeat-interval = 4 s
acceptable-heartbeat-pause = 20 s
}
Christian Sparre
@christiansparre
Feb 15 2016 07:04
@joonhwan I found the Remote.conf reference file to be helpful, it lists all the available settings and explains what they do. https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka.Remote/Configuration/Remote.conf
Vlad Kosarev
@vladkosarev
Feb 15 2016 15:17
Is there a way to check queue size and do something like a circuit breaker or just reject messages in that scenario? I am able to saturate my Persistent actor faster than it can process and the app runs out of memory and explodes (unprocessed messages are what's taking up all the ram).
Grzegorz Bernas
@profesor79
Feb 15 2016 15:54
tribute to: @Dantar for help in : TestinkFrameworkProblem
Arjen Smits
@Danthar
Feb 15 2016 16:01
:+1: glad to help
Jeff Cyr
@JeffCyr
Feb 15 2016 16:21
@vladkosarev I think you are looking for a NonBlockingBoundedMailbox. It's not yet implemented in Akka.Net, but you could create your own with something like this:
public class NonBlockingBoundedMessageQueue : IMessageQueue
{...}

public class NonBlockingBoundedMailbox : Mailbox<UnboundedMessageQueue, NonBlockingBoundedMessageQueue>
{...}
Vlad Kosarev
@vladkosarev
Feb 15 2016 16:22
@JeffCyr Thanks! I will check it out
Jeff Cyr
@JeffCyr
Feb 15 2016 17:28
@vladkosarev The messages are accumulating in your PersistentActor or in its underlying JournalActor?
alexgoeman
@alexgoeman
Feb 15 2016 17:30
Hi, is it possible for a parent to check the state of its childs , using the IActorRef it has cached during child creation ? Suppose child has become stopped but parent has still IActorRef cached, sending messages will not result in response,... Is overriding/implementing SupervisorStrategy or using Deathwatch the only mechanisms ?
Garrard Kitchen
@garrardkitchen
Feb 15 2016 17:48
Has anybody observed a "Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout" before?
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:01
@vladkosarev persistent actors have special rules, as they are stashing all incoming messages while recovering
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:03
@Horusiath so it's the stash that grows massive and blows up? what's the proper way to deal with that in case the actor is being spammed by messages?
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:04
the case is more general an its called fast publisher problem or something like that - when you have message producer, that is emitting messages far faster than consumer is able to process them
mailbox may flatten periodic spikes of activity, but won't help for constant overactvity of producer
and this is the reason, why backpressure was invented :)
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:08
right, I'm just not sure what the akka.net mechanism is for dealing with that. Ideally I'd have a circuit breaker on unprocessed queue size or something like that
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:10
until Akka.Streams will finally work in that context, there are few method to deal with that:
  1. Slower, but easier to implement - each producer message must be confirmed by consumer. Producer won't send a message, until ACKed previous one.
  2. Faster, but requires some more work - backpressure. In simplified version, when ready for receive, consumer sends a demand for producer. It's a number of messages it can consume. Producer stores demand counter, and when sending new message to consumer, decrements demand. When it reaches 0, producer won't send next message until consumer will send next demand message.
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:10
ouch :)
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:11
IMO circuit breaker is more for cases when component is so overused, that it crashes. What I proposed is a way to prevent it.
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:11
I see, more like a flow throttle
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:12
similar, but driven by capability of consumer's buffer, and controlled by consumer
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:13
I 'solved' it by adding some business rules for now (no basket should in reality have 100+ line items and so on) so even during load testing these rules do sanity checks and fail validation on the message spam past a point
obviously this won't work in analytics sort of scenario
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:15
before I forget: @alexgoeman Context.GetChild(childName) will return actor ref of living actor or ActorRefs.Nobody if no such actor exists. But if you're interested in particular actor, it's better to handle Terminated message, as it doesn't require active checking.
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:15
I think I can just make some sort of PressureCoordinator actor that will be the proxy and it's going to have to deal with logic of message overload
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:15
@garrardkitchen you may need to talk with @Aaronontheweb but more details will be necessary
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:17
what'd I do this time? :p
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:17

@Aaronontheweb

Has anybody observed a "Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout" before?

Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:17
hmmm
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:17
@vladkosarev it always depends on the use case
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:17
that's a new one - I'd need to see the HOCON
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:19
I personally like non-blocking buffered backpressure, but I'm porting Akka.Streams, so my PoV is little biased ;)
Vlad Kosarev
@vladkosarev
Feb 15 2016 18:20
nice :) I'm just trying things out with Akka and EventStore and that is messing with my head a bit because both these technologies allow for some interesting architecture
alexgoeman
@alexgoeman
Feb 15 2016 18:20
@Horusiath The "Terminated" message needs to be requested by using watch() ? or do parents always automatically receive Terminated messages from childs ?
Jordan S. Jones
@jordansjones
Feb 15 2016 18:21
@Aaronontheweb I’m the one seeing that error. I’m just curious if there was a “it is most likely this…” answer to the question :)
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:21
search the source code for the error message
and see what you find :p
Garrard Kitchen
@garrardkitchen
Feb 15 2016 18:21

this is not set in my hocon but looking at the source - https://github.com/akkadotnet/akka.net/blob/d00159ee8be45babecc7d20b935559df1415f49f/src/core/Akka.Cluster/Cluster.cs#L94 - and the error, it suggests the server is struggling to start the cluster system:

"Starting remoting"
2016-02-15 15:12:24 [Information] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 Remoting started; listening on addresses : ["akka.tcp://coach@10.100.4.198:9004"]
2016-02-15 15:12:24 [Information] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 "Remoting now listens on addresses: [akka.tcp://coach@10.100.4.198:9004]"
2016-02-15 15:12:24 [Information] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 Cluster Node ["akka.tcp://coach@10.100.4.198:9004"] - "Starting up..."
2016-02-15 15:12:44 [Error] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 "Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout'."
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Cluster.Cluster.<GetClusterCoreRef>d__10.MoveNext()
2016-02-15 15:12:44 [Information] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 Cluster Node ["akka.tcp://coach@10.100.4.198:9004"] - "Shutting down..."
2016-02-15 15:12:44 [Information] Version=6.0.0.77 Partner=Qualtrak MachineName=DBCTH-TEL-01 Cluster Node ["akka.tcp://coach@10.100.4.198:9004"] - "Successfully shut down"

pretty simple issue in this case
the actors literally weren't able to start up before that timeout
shouldn't happen very often, but looking at your logs it looks like a 20 second delay
Garrard Kitchen
@garrardkitchen
Feb 15 2016 18:25
I guessing I should RTFM and up the creation-timeout ?
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 18:25
@alexgoeman tbh I don't remember :/
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:25
I'm wondering how this could happen, because literally nothing on the network has started at this point
this stage has to occur before any of the gossip or cluster joining occurs
Garrard Kitchen
@garrardkitchen
Feb 15 2016 18:26
server load or possible thread sleep?
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:26
issue could be that the underlying dispatcher (ForkJoin) might have had an issue
since we did modify the custom TPL implementation in the last patch, although no one has reported any bugs with it yet
err
custom threadpool
if you're running under ridiculously high load
with a pegged CPU
that could do it
it'd have to be virtually spiked at 100% though
Garrard Kitchen
@garrardkitchen
Feb 15 2016 18:27
I've asked for CPU % util and also available RAM
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:27
in the meantime, could you file a bug for this?
Jordan S. Jones
@jordansjones
Feb 15 2016 18:27
Hmm.. could it be happening if the NICs are resetting?
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:28
we should investigate and see if it's an issue in our code first
before we rule that out
file a "possible" bug
Garrard Kitchen
@garrardkitchen
Feb 15 2016 18:28
will do. thx
Aaron Stannard
@Aaronontheweb
Feb 15 2016 18:28
@jordansjones could be - not sure to be honest. I've never seen this error on my own
Jordan S. Jones
@jordansjones
Feb 15 2016 18:29
No problem. Thanks for the brain dump on it though
Roger Johansson
@rogeralsing
Feb 15 2016 18:47
@alexgoeman you need to explicitly watch even from the parent in order to be able to observe the Terminated message in user code
Matthew O'Connor
@mattroid
Feb 15 2016 19:02
Hey all, I'm trying to persist messages with immutable lists, but I'm running into problems with deserializing the list from the peristant store. Is there an easy way to do this with out writing a customer deserializer?
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:02
@mattroid are you using default akka serializer?
Matthew O'Connor
@mattroid
Feb 15 2016 19:03
yes?
i just added the mongo persist plugin
and created a persistant acto4r
it persists and restores messages with normal types, but then I converted the message properties to immutables and now I'm getting ystem.FormatException: An error occurred while deserializing the Payload property of class Akka.Persistence.MongoDb.Journal.JournalEntry
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:05
Akka.Persistence.MongoDB uses it's own serializer - but tip for the future: don't relly on akka default serializer for persistence, it was not build for this purpose.
what is content of immutable list?
Matthew O'Connor
@mattroid
Feb 15 2016 19:06
list of strings
any recomendations for a good serializer? or just roll my own?
Aaron Stannard
@Aaronontheweb
Feb 15 2016 19:07
@mattroid use the Wire serializer
Matthew O'Connor
@mattroid
Feb 15 2016 19:07
ah thanks
that's the config for turning it on
for EVERYTHING though
not just persistence
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:08
@Aaronontheweb Wire setup how to is in the docs now
Aaron Stannard
@Aaronontheweb
Feb 15 2016 19:08
oh it is?
didn't know that
@mattroid hold on a sec. @Silv3rcircl3 mongodb persistence plugin used akka's default serializer or custom json based?
Marc Piechura
@marcpiechura
Feb 15 2016 19:10
@mattroid the mongodb plugin uses the normal bson serializer which can't serialize immutable collections by default
Matthew O'Connor
@mattroid
Feb 15 2016 19:11
good to know
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:11
general problem with immutable collections is that, nobody thought about serialization when creating them, so basically each immutable collection is special case in any kind of serializer
Matthew O'Connor
@mattroid
Feb 15 2016 19:11
so I assume it's trivial to tell the mongo plugin to use a differen t serializer
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:11
@mattroid I think, the best option would be to change immutable list thou
Matthew O'Connor
@mattroid
Feb 15 2016 19:12
@Horusiath you mean use the ImmutableList type?
because that's what Im using from the .net45 bcl librar
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:13
no, I mean use simple array instead - I know it's not safe, but currently serialization of immutable collections is grim task
Matthew O'Connor
@mattroid
Feb 15 2016 19:13
lol, i was wondering if that's what you mean
I built most of hte app with out immutable types and now i'm trying to upgrade to them
but maybe it's a pointless task, if we just treat our objects as immutable
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:14
I'm currious if mongodb .net maintainers actually know about this problem
Bartosz Sypytkowski
@Horusiath
Feb 15 2016 19:21
Lol, their github repo has closed issues section xD
Marc Piechura
@marcpiechura
Feb 15 2016 19:22
Man it's colder outside then I thought, iPhone shutdown while I was typing :D
@mattroid I have created serializer for immutable collections https://github.com/Silv3rcircl3/MongoDB.Immutable
But I'm not sure if they work with the current version, have created them a while ago and since I haven't updated the mongodb driver in my project I can't say if they still work
Matthew O'Connor
@mattroid
Feb 15 2016 19:24
I just ran accross that, but it looks like it depends on .net 5
System.Collections.Immutable
Haven't gotten devops to upgrade the build server yet, but it will happen soon, so it might be option soon
Marc Piechura
@marcpiechura
Feb 15 2016 19:26
my package? I think it's 4.5
Christian Sparre
@christiansparre
Feb 15 2016 19:27
Imuttable 1.1.37 has dep on DotNet 5.0 but 1.1.36 is still good with 4.5
Aaron Stannard
@Aaronontheweb
Feb 15 2016 19:27
we need to do a rollback for 1.1.36
have an open issue for that
.NET BCL team lost their minds when they changed the target numbers there
Marc Piechura
@marcpiechura
Feb 15 2016 19:28
Ahh ok doesn't know that, still using 1.1.36 I think
Matthew O'Connor
@mattroid
Feb 15 2016 19:34
i see
thanks i'll give it a go
Marc Piechura
@marcpiechura
Feb 15 2016 19:36
If it's not working feel free to create an issue and I will look into it
Matthew O'Connor
@mattroid
Feb 15 2016 19:36
will do
Bart de Boer
@boekabart
Feb 15 2016 21:59
Hi all. I'm struggling with a little something,
I created a dispatcher for Unit Testing, that allows the test to await the moment all dispatching (running, in fact) has completed
Now I can create this dispatcher in code (Sys.Dispatchers.RegisterConfigurator...) but how to do so in HOCON
(the cool thing here is of course that you'll have no more tests waiting for 3-5 seconds to complete, they all run instantaneously and reliably)
Bart de Boer
@boekabart
Feb 15 2016 22:16
Found it:
akka.actor.default-dispatcher.type = ""Akka.Vlow.TestKit.AwaitableTaskDispatcherConfigurator, Akka.Vlow.TestKit""
Bart de Boer
@boekabart
Feb 15 2016 22:52
Yeah, that's good. 350 actor tests now run in under 5 seconds (nCrunch)
I'll see whether I can make this into a nice contrib or testkit pull request, seems useful right?