These are chat archives for RBMHTechnology/eventuate

Jan 2017
Martin Grotzke
Jan 05 2017 16:46

Hi @krasserm and others! With an EventsourcedProcessor producing events for a replicated event-log the question is in which location it should run (to prevent duplicate, resulting events). There may also be other cases where, in reaction to some event, some higher level processing shall be done: e.g. in response to an OrderCreated event the order shall be analyzed and get a fraud-score - and because fraud-scoring is extremely expensive this should/must be done only once per order (this is a hypothetical example, another example could be to perform some payment transaction in response to an event). In both cases (the EventsourcedProcessor and the event-based, higher level "business process") the requirement is the same - the processing must be done only once per event.
Therefore either
a) the whole component (e.g. EventsourcedProcessor) is only in a single location active or
b) the component is active in all locations and decided per incoming event if it's responsible and should process or skip an event.

For a) one could configure in which location the component is active, some kind of "process-location pinning". This means that, if such a location is offline for a longer time, the configuration would have to be changed, so that processing continues in another location. To move the processing from one location to another would also require, that the processor in the new location knows, which events had already been processed, to start with the first unprocessed event. Today this problem would have to be solved by the application, AFAICS there's no such thing like a DistributedEventsourcedProcessor (and for a higher level business process it's the same - the new location must know the id/vectortimestamp of the last processed event).

For b) the consequence is that if a location goes offline for a longer time, it would not process events it's responsible for. Then another location would have to take over processing of its events. This would also require reconfiguration of some location so that it will take over the work of another location. And the challenge of a) exists as well: the new location needs to know which is the last event that was processed by the now-offline location.

These are just my ad-hoc thoughts on this topic, and we probably can bring it down to more basic principles in terms of consistency and consensus.

Do you have thoughts and maybe advices regarding this topic? Would it make sense that eventuate provides a general solution to this challenge, because it might be common for apps built on eventuate? Looking forward to your thoughts :-)

Martin Grotzke
Jan 05 2017 23:41
Btw, I just realized that both options a) Singleton and b) Sharding are slightly related to #143 (Akka cluster integration) - at least regarding the desired feature set.