These are chat archives for akkadotnet/akka.net

2nd
Aug 2015
Kyle Gobel
@KyleGobel
Aug 02 2015 17:19
Hey guys, I'm new to this actor model stuff, and have built a smallish app to handle some ETL pipeline things, I have some questions regarding if I'm doing things the proper way or not, would best place to ask about general (noob) things like this be on the github issues, or would you rather this be a stackoverflow type question?
Bartosz Sypytkowski
@Horusiath
Aug 02 2015 18:31
@KyleGobel I think, that here you can get the fastest answers. Please don't ask questions of the github, we use it for managing akka.net development.
Sean Killeen
@SeanKilleen
Aug 02 2015 18:38
@jcwrequests is there a specific roadblock that you think would happen with this? My guess is you could create x number of actors behind a router (round-robin sounds sufficient there), each of those would have a reference to its own SMTP actor within its context., and that SMTP actor would call SendAsync() in the appropriate fashion. You'd then add some supervision in case a given e-mail message blew up so that you could either notify your system if something was wrong with the message, or try again if the error was transient.
Does that make sense? (slash, everyone else double-check me? :) )
But I agree with @Danthar that you could just use non-async since all the actors would be processing in parallel and then just taking the next message from their respective mailboxes.
Kyle Gobel
@KyleGobel
Aug 02 2015 19:58

So I've got a setup almost like a bus.

My use case/problem
I need to do several things

  1. Read a JSON message from MSMQ
  2. Add Geo information to the JSON
  3. Add Device Data to the JSON
  4. Batch these messages up, waiting for more if need be (for example sake, say i'm creating batches of 1,000 messages)
  5. Compress and upload a batch of messages to Amazon S3

What I've done
I've created a seperate actor for everything: MsmqReader,AddGeoInfo,AddDeviceInfo,BatchMessages, UploadToS3. I then have a coordinator actor that just handles moving the data through this pipeline one message at a time. Every child actor sends its parent a Complete message when it's done, and the coordinator then sends the next actor the returned message, and so the message moves through the pipeline. (For the batching actor i'm using a scheduler that sends a CheckIfBatchReady message every so often to see if the batch has reached it's desired size, if it has it will sent the parent (the coordinator) a BatchReady message, in which case the coordinator will tell the UploadToS3 actor to get to work.)

My Question
I've just started learning about the actor model, and akka. I've done it this way because this is all I've learned so far, this seems to work and things are clean. I'm just wondering if I'm overlooking something or making some big mistake somewhere or something (I don't want to continue down this path if I'm just doing this flat wrong). I can share some code if that would help

Bartosz Sypytkowski
@Horusiath
Aug 02 2015 20:43
@KyleGobel looks like you've given a pretty much responsibility to a single actor (coordinator)
in this kind of scenarios, usually you'd like to have a stream-like functionality. In that case, it's the best to communicate between next stages/actors directly without coordinator
top level coordinator should be present mostly for handling eventual failures inside the pipeline
Kyle Gobel
@KyleGobel
Aug 02 2015 20:48
hmm, I thought about that, but then how would I handle re-use...i have another situation for example where steps 2-3 might be a little different (use different actors for those steps).
or in the actor world do you just not create 'generic' actors that handle a pretty generic task (like simply reading a message from msmq and sending back the result on completion)
Bartosz Sypytkowski
@Horusiath
Aug 02 2015 20:52
create common abstraction for building actor pipelines. Some basic protocol usually has message set similar to this:
  • Subscribe message to join actors working in the pipeline
  • OnNext message that wraps next message send from publisher to subscriber/s
  • OnError message send when an error occurred
  • OnComplete message to signal that no more messages will be send
this is actor equivalent of reactive extensions, and a part of something called Akka.Streams, we're working on (it's already available on JVM)
Kyle Gobel
@KyleGobel
Aug 02 2015 20:56
thanks for your pointers, i'll look into this
Kyle Gobel
@KyleGobel
Aug 02 2015 23:03
@Horusiath do you know of any examples of anything that does similar?