Hello all - it`s great to start doing it :smile: So final idea was summed up in issue comment, but I am posting it also here:
It is needed to implement the producer, which will support batch writes of events. It can be done by reusing parts of existsing producer or with Akka Actor + TCP.
Topic per journal
The write journal will write all events to partitioned topic, wheres the partitioning will be done by hashing persistenceIds. This will ensure that we will keep events in write-order per partition.
Recovery per persistenceId
This can be done in two ways:
Read per persistenceId
Same solutions as described above
The topic (view) should be created which will be feeded by background job. The job will read all events from main topics and push new persistenceIds if sequenceNr of event is equal to 0.
The topic (view) should be created per eventType which will be feeded by background job. The job will read all events from main topic partitions and copying the tagged events. With this approach we can ensure, that eventsByType view will contain events totally ordered by partition (almost persistenceId), but it will not ensure that we have total order of events between partitions.
The background job mentioned above should start on each node, which will request any of query stream, eg. allPersistenceIdQuery or allEventsByTagQuery. It also can be configurable to not start this job, if user do not want to. Those jobs will use default Java consumer and will belong to same groupId which ensures that we will not read same event on more time.
Here https://gist.github.com/maciekciolek/c645e28cc708b26177eb1444d077d09a is the snippet of code, which allows produce batch of events per single partition.
From my experience current Java client is not an extensible piece of code :worried: I am not certain if we should base our producer on their code, mayby we can use only message formats and implement producer with Akka TCP (http://doc.akka.io/docs/akka/2.4.7/scala/io-tcp.html).
@Fristi as I remember you proposed to share Kafka`s protocol defined int scodec?
@all what do you think about idea of implementing batch producer for Kafka with Akka TCP?
Hello everyone, I want to ask you what will be a good usage of kafka as storage backend for eventuate? You have a use case already?
I was interested in Kafka because it can work as an Event Bus, allowing me to persist an event and publish it to the event bus in an atomic operation. So Kafka would be acting as the storage backend for my eventuate-based microservices and the mechanism to communicate the different microservices as well.
However, I kept reading eventuate documentation until I finally realize that Eventuate is Akka on steroids: it not only allows AP, it also gives you the event bus with a lot of features.
These actors can be considered as event-driven and event-sourced microservices, collaborating on a causally ordered event stream in a reliable and partition-tolerant way. Furthermore, when partitioned, they remain available for local writes and automatically catch up with their collaborators when the partition heals.
The only constraint is that all our microservices must be eventuate-based (this doesn't mean that I must take the DDD+Actor approach), so I think that if we want to communicate a bunch of polyglot microservices, it might be a good use case for Kafka as storage backend.
In my case I think I will try with all my microservices with Eventuate and Cassandra is good enough because I don't have any requirement for the storage backend beyond fault-tolerance and high availability.