These are chat archives for RBMHTechnology/eventuate
Global-scale event sourcing and event collaboration with causal consistency
@Tvaroh I thought a little bit about your application. I still think it is hard to come up with a concrete proposal based on what you have written, but I will try to give a few hints that might be worth thinking about.
As far as I can tell your are basically struggling with two things: How to use a Neo4j backed write model and how to automatically resolve conflicts in case of concurrent and conflicting updates.
First of all currently eventuate does not really support persistent write models. So if you still want to use Neo4j as your write model and eventuate to distribute the changes to your Neo4j in form of events you could perform your command validation and database updates outside of eventuate and persist eventuate events based on the applied changes. You typically want to do this in a transactionally safe manner, i.e. for each successful transaction write corresponding events (even if your application crashes right after a successful commit) but do not write events for failed transactions. If Neo4j allows to access its transaction logs it might be possible to read the changes from there, persist corresponding events along with an event that indicates which transaction has been processed successfully, so the application knows where to continue after a restart (This is basically the CDC Martin mentioned above).
An alternative is to use eventuate for initial command processing, but then your write model must be in-memory. In that case you might want to rethink what is actually needed to validate your commands. Maybe you do not need the entire graph for this, but just some aspects of it. Maybe you can partition your data and use
aggregateIds in your
EventsourcedActors for example using user-ids as
aggregateIds. If your application allows this you could validate your commands based on this in-memory write model and use an
EventsourcedWriter to populate your Neo4j as query database.
(From my point of view) one of the core features of eventuate is to detect concurrent updates and allow to resolve conflicts of such updates based on application defined (i.e. high level) events (as opposed to using generic (low level) CRUD events on table-row basis for example). So it is up to the application to define and persist events in a form that conflicts can be resolved in a way that makes the most sense for the application. You basically have to built your application specific CRDT. As far as I understand your central data structure is a tree so maybe it makes sense to read the section 3.4 about Graphs in the paper A comprehensive study of Convergent and Commutative Replicated Data Types and build upon its ideas.
Just my 2c. I hope this helps, but maybe someone else has better ideas. I assume in any case it is going to be hard to solve all the details through discussions in this forum :-)
@Tvaroh @volkerstampa Great thoughts! I also thought a bit about the application, but couldn't write my thoughts due to bad connectivity. Now here they are:
After all to me it seems rather complicated to get this right/correct and robust. Perhaps it's worth it evaluating an open source / free graph database that supports replication, as e.g. OrientDB (multi master, quorum based strong consistency).