These are chat archives for akkadotnet/akka.net

16th
Aug 2018
tiny hydra
@tinyhydra
Aug 16 2018 00:56
what about - is there a way to limit how many actors of a specific type can be created before they're reused?
related: can actors be reused automatically?
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 06:23
@Lutando what's wrong with fallbacks in your case?
@tinyhydra what do you mean by reused?
Lutando Ngqakaza
@Lutando
Aug 16 2018 06:58
I want to have Config B not fallback to Config A, rather to the akka defaults
Vladimir Luzhin
@VladimirLuzhin
Aug 16 2018 08:51
@Horusiath Hi, can you tell me, please, can I somehow get data in the controller without ASK? I will have about 10,000 thousand requests on the controller every 5 seconds and if I use ASK, everything will crash.
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 10:35
@Lutando what's for? You can just stop using these config values in config A. I don't recall any configuration format that works like you want to.
@VladimirLuzhin how does it crash?
Sam13
@Sam13
Aug 16 2018 11:15
Question about at least once delivery: Is there any built-in functionality to configure that only a certain number of messages is tried to be sent? I analyzed the current code and found that each message is sent for the first time and if not confirmed it will be redelivered within the redeliver functionality.
If the receiver of those messages is very slow and many messages are generated it will slow down the receiver. It may not be able to confirm all messages and a lot of them need to be redelivered which will slow down the receiver even more...
So what I want to do is e.g. to queue messages which will be sent when the unconfirmed count reaches a certain threshold
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 11:31
@Sam13 you can setup akka.persistence.at-least-once-delivery.warn-after-number-of-unconfirmed-attempts to make your actor send an UnconfirmedWarningmessage to itself if some message cannot be delivered for some reason.
Sam13
@Sam13
Aug 16 2018 12:37
@Horusiath What's the benefit of that? The goal I want to reach is that the target actor (which lives on a remote side in another actor system) does not get overloaded with messages. The current implementation tries to send any at least once message directly which may produce a lot of messages and slow down the target actor (which causes redelivery). I want to limit the number of messages being sent to avoid that.
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 12:56
@Sam13 currently redeliveries are send every 5sec to a single actor - if this is too often, you can change it, but it looks more like an issue somewhere in the design or processing logic.
Sam13
@Sam13
Aug 16 2018 13:43

@Horusiath Thanks, I'm aware of that setting. Maybe I miss something but as far as I understand the current implementation it's not possible to prevent "overflooding" the "target" of an at least once delivery actor.

Assume we have an at least once delivery actor which forwards messages to a remote actor of another actor system.
We have a constant message rate produced from the at least once delivery actor.
The target actor will initiate some actions based on it's received messages which may slow down it's message acceptance / confirmation rate.
If it's message acceptance rate gets very slow the at least once delivery actor continues sending with it's original message rate and on top of that all message which have not been
confirmed (because of slowed down message acceptance / confirmation rate of the target actor) will be sent again.
This means the message rate from the at least once delivery actor will increase and the load on the target actor increases as well.

These are my observations, so I'd be interested about your thoughts or if I miss something here...

mwardm
@mwardm
Aug 16 2018 15:40
@Sam13 I think you're mixing up the need to guarantee delivery with the need to do the work. You want the target remote actor system to acknowledge that it's received the message ASAP. (It's then that system's responsibility to ensure the message doesn't get lost from that point.) Then the remote system can do the work taking however long it wants, presumably returning the results when it's done. Probably easy to implement with a pair of actors on the target system, the first of which just receives messages, acknowledges receipt to the source actor system, and forwards the message to another actor (like your existing one) that actually does the work. Maybe?
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 16:47
@mwardm nope - what you're usually interested in is to process the message. The fact of delivery itself has no value if it was not processed (and this could be a thing i.e. when receiver confirms, that it got a message before processing it but the machine hosting it crashed right after sending confirmation).
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 16:52

@Sam13 what you're talking about seems to be more the problem of how are you handling the messages - in most of the standard cases there should never be a need to send the message more than once. This should happen only if message was lost due to network split or machine crash. Otherwise you should think about relaxing redelivery interval or parallelizing the work.

Also if your computations are expensive (hence there's any risk of not emitting confirmation on time), try to think about some mechanism for deduplication - so that, when the message that was processed before is received twice or more times, it won't be processed again and again. Instead if it was handled once, respond with confirmation right away.

mwardm
@mwardm
Aug 16 2018 17:29
@Horusiath It's a bit semantic, but I'd argue that delivered and processed are two different things that don't need to be combined and in this case (because of the difficulties it's causing) shouldn't be. The flag is called at-least-once-delivery :-) Whether or not the message has been functionally (successfully) processed can be indicated by a response message returned to the source system at the end of processing. (If you need to check it's actually been processed e..g within a given time, then you can you can implement your own mechanism for that - set a timer message for an hour or something. There's no point in resending a message that was successfully received and is just waiting in a queue.)
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:32
https://petabridge.com/blog/how-to-unit-test-akkadotnet-actors-akka-testkit/
"How do I change the configuration of the TestActorSystem?" -> if I understand this correctly, the default level is DEBUG. Is that correct?
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 17:33
@mwardm what is the value added of message being delivered but not processed?
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:33
I just changed actor's _log.Info into _log.Debug and updated the test to use EventFilter.Debug, however, it is failing as no matching messages received
but if I add : base(@"akka.loglevel = DEBUG") to the test class, then it passes
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 17:34
@snekbaev I'd set up loglevel explicitly if you really want to test it with event filter
by default logging level is info for most of the frameworks and platforms I can think of - the same case for default akka config
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:35
@Horusiath any ideas why DEBUG is not default in tests?
ah, hm. However that article says to mute the debug messages, as it was default :)
mwardm
@mwardm
Aug 16 2018 17:37
@Horusiath He's sending the message to a remote system. You're ensuring it gets there reliably.
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 17:38
@snekbaev for time-sensitive tests loglevel can change the actual behavior - I've seen many racy tests that were failing when INFO level was set, but they passed on DEBUG level thanks to delays introduced by printing debug statements.
mwardm
@mwardm
Aug 16 2018 17:39
When I send my returns to Amazon, I make sure it's sent Signed For so I know they got it. If they don't refund me, I call them in a couple of weeks and I've got proof that it's their problem :-)
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:39
@Horusiath doesn't that mean we should have some other ways of handling those racy conditions than relying on the loglevel to throttle it
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 17:40
@mwardm so what? ;) Actor confirmed that message was received, not that it was processed - this is how lazy coworkers set up false expectations in my previous jobs ;P
@snekbaev I don't rely on loglevel to pass my tests, I'm just saying that principle of observer changing the behavior of observed object also applies here. For that reason I always try to keep my configuration possibly closely to what I expect on production.
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:42
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 17:44
@snekbaev do you have tests running in parallel?
Shukhrat Nekbaev
@snekbaev
Aug 16 2018 17:47
shouldn't be, when I run all I can see them going one by one, unless Resharper is doing some magic with nunit3
mwardm
@mwardm
Aug 16 2018 17:49
@Horusiath But most of the time in Akka you send a non-guaranteed one-way request message and when the target actor has processed it it sends a non-guaranteed one-way message back as a response, don't you? (Maybe I've been doing it my own way!) That's how you implement the logic of the system. I don't see a need to change that mechanism just because the actors are remote - but it's prudent to add in an extra check that the messages actually got across the network, which is what the at-least-once-delivery adds.
@Horusiath Arguing with myself a little... I'd actually...
mwardm
@mwardm
Aug 16 2018 18:00
...rather that code wasn't required to acknowledge the at-least-once-delivery. In my mind, at-least-once-delivery ought to be an infrastructure / deployment time thing. (I haven't used Akka in a few months so I'm a bit hazy on the details...) Given that code is required, your "receipt-message-means-it's-processed" message is an equivalent mechanism - except that you've got infrastructure getting in your way if the processing takes longer than 5 seconds.
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 18:02
@mwardm
  • in 90% of the cases you can rely on single at-most-once delivery, possibly combined with setting up a persistent queue or log (like kafka or rabbitmq) in front of your actor system - if message was not delivered on any point, while it should be, you simply retry the entire pipeline on the entry level or event at client level (i.e. by waiting for response to timeout).
  • in most of the remaining cases, usually setting up a custom rety method - like scheduling retry tick or even using Polly if you don't really care about squeezing performance - is enough. Sure actor can die an messages get lost, but this can be done on architectural level or simply as part of the business process.
  • at-least-once delivery is really super rare requirement, if you cannot use persistene queue for this part of the pipeline for various reasons, and you want to be absolutely sure, that delivered message has been processed (singe ALOD relies on persistence, so redeliveries survive both sender and receiver deaths)
mwardm
@mwardm
Aug 16 2018 18:11
@Horusiath Okay. I'm still happy my method was an easy and logical fix for the problem while at-least-once-delivery was on and using default settings. [In my dreams, there's a day in the distant future where I'd love to have a pit-of-success discussion on some of the stuff above!] But I think what you just posted says the OP should turn off at-least-once-delivery and just implement his own retry mechanism. Which works too. Cheers.
Bartosz Sypytkowski
@Horusiath
Aug 16 2018 18:16
yep, I think I did :P
Vladimir Luzhin
@VladimirLuzhin
Aug 16 2018 23:00
@Horusiath after 10-20 requests start throwing AskTimeoutException