These are chat archives for RBMHTechnology/eventuate

13th
Sep 2016
Mark de Jong
@Fristi
Sep 13 2016 18:18
Untitled.png

I'm tinkering awhile with this idea to use Kafka and Akka cluster sharding to do Event Sourcing/CQRS.
It's a alternative approach to akka-persistence like Eventuate. I wonder if my approach would be feasible and if it's similar to Eventuate.

All writer nodes join a akka cluster and route the commands to the node which runs the aggregate actor. The actor will write this atomically to Kafka in one produce call (see flumina which makes that possible)

  • the events produced after validating the command to the current state of the aggregate to maintain the aggregate invariants
  • the state after the events are applied to the current state.

The events are written to a events topic which can be consumed by multiple nodes and different types of microservices to create autonomous and a event driven architecture. Offsets of consuming these topic are either stored in Kafka or a local KV database (favoring the last though).

The state is written to a log compacted topic. Each node will receive these state updates and will write the state in a local KV database.

When a node goes or a persistent actor is spawned on different node it will load the state from local KV database and start operating on incoming commands + events.

WDYT? Any pitfalls? Similar to Eventuate? :-) Happy to have some feedback
Martin Grotzke
@magro
Sep 13 2016 18:34
@Fristi Just wondering, what kind of consistency do you want/need?
Mark de Jong
@Fristi
Sep 13 2016 18:37
Aggregate state should be consistent and partition tolerant. Read models are more relaxed however as they are more focused on availability.
The atomic write of events + state is good, only thing which can cause major problems is if Kafka won't deliver the states to the nodes at all or not in time (if a actor gets passivated and spawned on a different node which didn't caught up reading the states)
Martin Grotzke
@magro
Sep 13 2016 18:40
I haven't worked with kafka, but AFAIK it provides sharding + replication. You're probably going to use only replication for the event log, right?
What happens in case of a network partition, can kafka replicas diverge?
Maybe the question is not important...
Martin Grotzke
@magro
Sep 13 2016 18:48
What I'm finally after is the following:
  • assume aggregate A1 emitted event E1
  • aggregate A2 consumes E1 and emits E2
    Does kafka guarantee, that any client receives E1 before E2? Or is this even not important for you?
Mark de Jong
@Fristi
Sep 13 2016 18:52
Good questions. I know that per topic the partitions are ordered. However I haven't read up on network partitions etc. I've read Kafka is CA
Martin Grotzke
@magro
Sep 13 2016 18:52
I think it depends on the business requirements which kind of consistency is needed. I think the think which differentiates eventuate from different solutions is its support for causal consistency. If you need this, you'd have to build it on your own, but then you're probably solving things again that eventuate already solved.
Mark de Jong
@Fristi
Sep 13 2016 18:53
The last question is about ordering, but I am not sure if I understand correctly. A1 = aggregate with id 1 and A2 is aggregate with id2 ?
Or is it a instance of the same entity ?
Martin Grotzke
@magro
Sep 13 2016 18:54
Being CA means that there cannot be network partitions :-) You have to choose between C and A if you accept that there can be network partitions...
I thought "A1 = aggregate with id 1 and A2 is aggregate with id2".
Mark de Jong
@Fristi
Sep 13 2016 18:56
I see. Have to take a closer look to that.
Martin Grotzke
@magro
Sep 13 2016 18:57
I didn't read if fully until now, but this might be a good read: https://aphyr.com/posts/293-jepsen-kafka - might be a bit outdated, but the basic concepts should still be the same. Of course the official documentation should answer the important questions :-)
Mark de Jong
@Fristi
Sep 13 2016 18:57
If they A1 and A2 are different entities the order of A1 emitting a event doesn't matter for A2 emitting events?
Good to be critic about Kafka! Thanks
Martin Grotzke
@magro
Sep 13 2016 18:58
Say there's also A3 - would it be ok for your application/business, if A3 receives E2 before E1? That was my question.
Mark de Jong
@Fristi
Sep 13 2016 18:59
The events should be ordered
Martin Grotzke
@magro
Sep 13 2016 18:59
But according to the linked post kafka should be CP, so this shouldn't happen.
Right.
Instead, in case of a network partition between replicas kafka would not be available if it's fully CP - not sure if they have new concepts like quorum or stuff.
Might also be interesting to go through: https://issues.apache.org/jira/browse/KAFKA-1555
Of course for more serious support on such questions related to kafka it would be better to ask the kafka community :-)
Mark de Jong
@Fristi
Sep 13 2016 19:06
Yea of course, wondering about my own little distributed problem with storing aggregate state in local KV store.. what if node goes down and the node which takes over hasn't caught up yet? I was thinking about adding a version number to the state of aggregate and each command has to include the version of the state they want to apply the command to. That would solve the issue of having inconsistency.
Martin Grotzke
@magro
Sep 13 2016 19:17
Afk now, reading later...