These are chat archives for RBMHTechnology/eventuate

26th
Oct 2016
Dan Di Spaltro
@dispalt
Oct 26 2016 06:23
Why does the cassandra log adapter need to deal with partitions on the read side? Wouldn't the events come out in order without specifying the partition?
Martin Krasser
@krasserm
Oct 26 2016 07:01
Yes, reads from different partitions are concurrent
Dan Di Spaltro
@dispalt
Oct 26 2016 15:32
ahh so it's just a concurrency thing?
Martin Krasser
@krasserm
Oct 26 2016 16:18
yeah, local logs have a total order that must be preserved when reading
Dan Di Spaltro
@dispalt
Oct 26 2016 16:48
sorry let me back up for a sec, I am writing an log adapter, I am looking at the cassandra one for inspiration. I am not sure why on the read side you iterate partition by partition. Particularly since the pk is partition, seqnr, wouldn't just pulling it all out sequentially (a whole table scan) achieve the same thing versus iterating each partition?
Or is the seqNr scoped on a partition?
Martin Krasser
@krasserm
Oct 26 2016 16:53
The whole table is partitioned with partition as the partition key. When reading the whole table the driver reads from the partitions concurrently. You can see these effects in http://rbmhtechnology.github.io/eventuate/reference/adapters.html#batch-processing for example, where the Cassandra connector is used to read the table
So, to get a total order, you need to read the partitions sequentially. Cassandra doesn't do that for you i.e. doesn't support sorting by partition key, only by the clustering column (= seqnr) within a partition.

Or is the seqNr scoped on a partition?

no

What storage backend do you plan to support?
Dan Di Spaltro
@dispalt
Oct 26 2016 16:57
vitess.io
it already has a concept of partitions built in
not entirely sure how to rectify that
but Ill figure it out!
I'm pretty close to a toy working version
the code kinda looks like shit though
ill post the repo here when I get a full working example
Martin Krasser
@krasserm
Oct 26 2016 16:59
please do that, thanks!