These are chat archives for RBMHTechnology/eventuate

13th
Oct 2015
Magnus Andersson
@magnusart
Oct 13 2015 04:49
Hi I have been thinking about how to best model situations where a command needs to be validated on a system level. Example: account creation,
Magnus Andersson
@magnusart
Oct 13 2015 04:56
I need to verify the account email does not exist from before, there could be multiple similar validations. Would you model that as a two distinct events? First verify at one place where you have the informationinformation about occupied emails on a system level and reserve the email as an event. Then create the actual account as a separate command/event that arise as an outcome from the previous step.
I'm looking at the order example where no such validations take place/are needed. The conflict resolution phase is what is tricky because during a split brain the conflict in my example is not on singular account level but on system level and the event needs to be recorded on that level?
Mike Mazur
@mikem
Oct 13 2015 06:20
@krasserm cool!
Reading through the endpoint docs made me think about replication. Clearly eventuate can be configured to replicate events between nodes. What if we have Eventuate write events to Cassandra, and configure Cassandra to handle replication for us? I imagine that events written by one Eventuate node will appear in the store at other nodes. Will Eventuate discover those events and apply them locally to bring its state up to date?
Martin Krasser
@krasserm
Oct 13 2015 06:37
@mikem Eventuate guarantees causal storage order which can only be enforced on top of Cassandra, hence the Eventuate-specific replication between locations. Cassandra replication within a cluster can still be used within a location but this is for achieving stronger durability. More about bolt-on causal consistency in the very good 3-part article series of Adrian Coyler (links in this tweet of Peter Bailis https://twitter.com/pbailis/status/643458300294598657)
Mike Mazur
@mikem
Oct 13 2015 06:39
@krasserm ok, great, thanks, that clears it up
mikem @mikem follows links to the blog posts
Martin Krasser
@krasserm
Oct 13 2015 09:11
@magnusart I'm not sure I understand your question. Do you want to know how to enforce a global invariant (such as a unique email address)? And, what do you exactly mean by recording events on system level?
Magnus Andersson
@magnusart
Oct 13 2015 13:53
@krasserm Yes I’m asking about enforcing global invariants when using the pattern where each aggregate root is an actor with a cRDT.
In the case where two conflicting accounts were to be created during a network split. Let’s say I wish to disable the accounts until a manual conflict resolution have been applied. I’m making the assumption that this type conflict resolution would belong in a separate location (ie "system level”) rather than putting this logic into the account actor. I’m looking for a sane way to elevate this conflict resolution since I believe it is not part of the account life cycle but of the life global cycle of the system/application. But maybe that is just me not having a clear picture of how these issues are typically resolved.
Hope that is marginally clearer. If you wish you can take the eventuate order example and assume only maximum 10 orders can be in an active state at one time or some other silly rule that enforces an application global limitation. Where does conflict resolution (cRDT) belong in the context of eventuate?
Martin Krasser
@krasserm
Oct 13 2015 14:34

You have two options to deal with global invariants:

  • Enforce global invariants with global coordination. For example, you could delegate account creation to a single location. A variant of this is using a global coordination service (not part of Eventuate) to elect a leader location. The price for global coordination is limited availability.
  • Allow temporary global invariant violations and resolve them later. For example, a global invariant in the example application is that orders must have a unique id. Concurrently created orders with the same id are detected as conflict. The example application requests the user to resolve that conflict interactively. Other applications could also resolve such a conflict automatically (see user guide for details).

I think the “system-level” involvement you mentioned is the required global coordination to enforce invariants. Does that help?

Magnus Andersson
@magnusart
Oct 13 2015 14:49
Yes, I’m on the second option. But I’m trying to figure out where to put the responsibility on an implementation level. In the case of a unique order id each order actor only knows about it’s own order ID, therefore it cannot detect duplicate orders on the order actor level. So the the detection mechanism for this I suppose could be a repeat attempt to create a order actor in the Manager when replaying the event? I was assuming I needed a second cRDT for detecting a conflict among the unique orders but perhaps that is not the case?
Martin Krasser
@krasserm
Oct 13 2015 14:59
This can be resolved on the level of order actors because both (concurrently created) actors consume each other's created event.
Magnus Andersson
@magnusart
Oct 13 2015 14:59
Right, but what if I say 10 orders active at one single time?
(I’m not being obnoxious on purpose, I just want to make sure I understood)
Martin Krasser
@krasserm
Oct 13 2015 15:12
Then you need to enforce that on OrderManager level. Really good questions BTW, keep them coming :smiley:
Magnus Andersson
@magnusart
Oct 13 2015 15:14
I find it hard to reason about concurrent writes when you have business logic which transitions through states based on certain preconditions in multiple levels, when the basis for the conditions can be changed ”after the fact". If you’re in the transactional-single-machine mental model it is quite straightforward. In the distributed model it is not by any means impossible to model, but it is hard to get a clean model where you can easily see what has been affected by and needs to be addressed during a conflict resolution.
Martin Krasser
@krasserm
Oct 13 2015 15:16
Yeah, things are easier with strong consistency ...
Anyway, it's absolutely fine to use coordination where needed and use weaker consistency models for other cases. Many applications use both.
Magnus Andersson
@magnusart
Oct 13 2015 15:22
Let’s say you create the account and the duplicate users creates a bunch of content in each node during the net split and order products with different credit cards. Perhaps not the most realistic scenario, but it illustrates the problem. Something needs to be done of course, it’s a mess and probably somebody needs to look into it manually. I’m trying to reason around a model/pattern to cleanly pick up the thread and trace through the system what happened and then based on that apply different strategies to resolve the conflict. I want to put this logic in it’s own special hell, not in the account actor :).
But maybe you just gave me the answer, don’t make it so complicated ;). Relax a bit and have coordinaton points.
Martin Krasser
@krasserm
Oct 13 2015 15:26
A more formal approach to that topic is described in http://www.bailis.org/blog/when-does-consistency-require-coordination/ and the linked paper. Great read.
Magnus Andersson
@magnusart
Oct 13 2015 15:26
Cool, thanks. I’ll have a look!
Martin Krasser
@krasserm
Oct 13 2015 15:28

I want to put this logic in it’s own special hell, not in the account actor

Use a hell-actor for that ;) (comparable to the 10 open orders invariant maintained by the OrderManager)

Magnus Andersson
@magnusart
Oct 13 2015 15:30
Yes, hell actor it is. :D Thanks for answering my questions, if it gets too much off topic please tell.
Martin Krasser
@krasserm
Oct 13 2015 15:35
Not at all, great discussion!