These are chat archives for atomix/atomix

16th
May 2017
Ragnar Rova
@rrva
May 16 2017 17:53
if I would like to replicate a dataset between java processes (let's say 100k items each 1k in size) , and the replication is single producer possibly multiple consumers, but needs to be resilient of consumer failures, like restarts, message loss etc, and needs to only use HTTP as transport, is copycat something to look into?
Ragnar Rova
@rrva
May 16 2017 18:04
it seems to me that raft is overkill, since there is always a single producer, but I cannot find any readily implemented replication algorithms for in-memory data replication over HTTP for java processes
Jordan Halterman
@kuujo
May 16 2017 18:17
It's absolutely overkill. You need HTTP Kafka :-) ...which I suppose doesn't exist?
Ragnar Rova
@rrva
May 16 2017 18:19
yes, HTTP Kafka with log compaction, or better yet to be independent of kafka long-term resilience some way to say "give me a snapshot of current state, and then I will start a subscription from that point"
but preferably with having to implement as little as possible myself in producer and consumer code
Ambud
@ambud
May 16 2017 18:23
Yes it's a typical Kafka use case. You can use the Kafka rest proxy to go over http
Ragnar Rova
@rrva
May 16 2017 18:23
I guess that's what I want then...
thanks
only problem is that there is reliance on another external entity for the replication (kafka)
This is also on option if you want to embed this in your producer however fault tolerance is going to be an issue if you embedded
If you message volume is extremely low like 1-10/s Kafka is an Overkill
Ragnar Rova
@rrva
May 16 2017 18:27
it is that low
Ambud
@ambud
May 16 2017 18:27
You should look at something like Apache ignite based queues
But you don't get checkpointing like Kafka with it
Ragnar Rova
@rrva
May 16 2017 18:28
the entire dataset (100k entries), does not receive more than a 1k updates per 24 hours, but reaction time to updates needs to be low, lets say <10s
well 10s is not low, but anyway
Ambud
@ambud
May 16 2017 18:30
Kafka is the quickest solution, everything else will need work. You could also look at any mq based system
Ragnar Rova
@rrva
May 16 2017 18:30
I like kafka, having worked with it. I just need to look into all the HA issues that need to be adressed..
Ambud
@ambud
May 16 2017 18:31
To make sure this discussion doesn't spam the atomix user feel free to DM