These are chat archives for atomix/atomix

28th
Sep 2016
Jon Hall
@jhall11
Sep 28 2016 00:19
any thoughts on another rc?
smillies
@smillies
Sep 28 2016 10:36
Hello there, I'm new to Copycat and still trying to figure out some things. Currently, I'm investigating how partitioning works in Copycat. My aim is segmentation by key, and maximizing concurrency on that. It seems that Copycat server is essentially sequential, see the discussion on August 8. But on the other hand, in this post for example, Jordan Halterman talks about a partitioning model for Copycat, but I have found nothing in the API on how to use it.
Jordan Halterman
@kuujo
Sep 28 2016 17:13
There has long been discussion about partitioning in Copycat and Atomix. But it has been avoided to instead focus on stability and the correctness of the Raft implementation. Introducing partitioning into the Raft logs for Atomix resources requires the addition of things like 2pc to coordinate across partitions to e.g. create/delete resources. Additionally, to make partitioning properly scalable, Raft's leader election algorithm has to be modified to prevent hotspotting where the same leader is elected for all or most partitions. It is because of those complexities that this has not been implemented, though the API is actually capable of allowing a user to set up a partitioned cluster. In order to do that, one would just have to build a Transport that can be shared among multiple Copycat/Atomix instances.
smillies
@smillies
Sep 28 2016 17:51
Thank you for the explanation. I was wondering what the issues were in implementing partitioning. With regard to setting up a partitioned cluster by sharing a transport between multiple Copycat instances, could you be more specific how one would do that? For example, with the Netty transport?
Jordan Halterman
@kuujo
Sep 28 2016 18:05
Yeah, partitioning is easy if you don't have to have transactions across partitions. If they were implemented, we'd want to be able to create and delete Atomix resources, which would require 2pc to make sure all creates/deletes are applied or none are, but even 2pc isn't perfect
So...
Jordan Halterman
@kuujo
Sep 28 2016 19:04

This is actually why we separated the transport. The idea is, you'd have to create a wrapper around NettyTransport that allows multiple Copycat instances to share the same Transport and the same Connections within them. When a server creates a new connection, if a connection already exists to the Address the existing connection is returned. There's also nothing stopping you from creating literally separate Transports, it would just require a separate port per partition.

But the other challenge with partitions in Raft systems that implement sessions like Copycat does is for each partition there's going to be some additional overhead for the session. This is the other problem that's challenging to solve. For each partition, in order to maintain linearizable semantics the client has to send period keep-alive requests. This limits the total number of partitions you can have since there's additional overhead for each partition. It also complicates how to handle session failures. It's entirely possible a session could be lost on one partition but not the other, in which case linearixability is lost on one partition and not the other and writes can complete on one partition but not the other.