These are chat archives for atomix/atomix
DistributedValueresource (doing cas operations). @jhalterman is currently working on integrating configuration changes into the Jepsen tests (arbitrarily adding and removing nodes while writing to the cluster and compacting the logs). So, that's all to say up until about a week ago after I finished support for snapshots, all the effort was put into algorithms and features and handling various fault tolerance issues. But snapshots represented the end of that work, at least in Copycat. All we're doing now is testing Copycat/Atomix as much as possible, and it has been and I expect will continue to become much more stable every day. We'll push a release candidate once Jepsen testing with configuration changes is complete, and a full release if there are no issues with the release candidate. But Atomix will continue to be informed by use cases. Copycat/Atomix makes it incredibly easy to add new tools like a work queue.
DistributedWorkQueuein Atomix is to have something that will push work to clients rather than require them to poll and to guarantee items are not lost of a node crashes. So, what the work queue state machine does is: when a task is committed and applied to the state machine, it will push the task to a worker (client) if there's one available. Clients/replicas acknowledge a task and get the next task in a single commit. So in that sense it will be much faster than polling a queue, but two commits are still required to process one task (put the task on the queue, acknowledge the task). We could probably make a much more efficient version that acknowledges tasks in batches though. As for threading, in
AtomixReplicathe user always receives messages in a different thread than the server, and always in the same thread. The reason for using the same thread is because session events can be linearizable, so handling a message on the client has to be synchronous to maintain linearizability. But we could make a work queue that allows a single node to process a lot of tasks concurrently and ack them in batches since the task queue could be asynchronous anyways... If that makes sense. I guess there are several ways to go about it, but the scalability limitations of majorities in Raft will be there at least for writes to the queue.
DistributedMembershipGroupto do cluster management, allow clients to discover servers, and allow a leader to communicate with followers, a
DistributedLeaderElectionto coordinate and assign topics and partitions, and a
DistributedMessageBusto send messages from clients to servers and synchronously replicate between servers according to the topic's replication factor. I expect in the future that use cases of will inform the development of more complex Atomix resources to make things like that easier, but for now Atomix resources will just remain low-level enough to allow people to build things that we can perhaps turn into resources in the future.
DistributedMembershipGroupto track the set of nodes that are alive. A node will be removed from the group if it crashes. Then, on each node use the
DistributedMessageBusto receive messages on that node. Clients would use the same
DistributedMembershipGroupto determine where to send messages, and individual messages can be acknowledged by periodically updating a
DistributedLong. This I think could be one of the common patterns that's turned into an Atomix resource to allow for faster reliable messaging. But the problem with high throughout inside of the Atomix cluster - in a state machine - is everything is limited by memory right now, and so messages can't back up too far. If the client is reading from a persistent store and can replay messages then I guess that's not an issue... We'd just have to implement some sort of back pressure mechanism. I guess it wouldn't be too hard to create a resource that does that actually - faster reliable messaging from a persistent store - if the process pushing messages is reading from disk or something.
DistributedMessageBusis unreliable insofar as it's the sender's and receiver's responsibility to coordinate to ensure messages aren't lost. It would be actually fairly easy to provide some reliability in the sense that messages will be automatically resent if lost, and it could even be feasible to do in-memory replication of messages by coordinating the replication through Atomix (
DistributedMessageBusbasically uses something like a membership group internally to coordinate messages already), but that wouldn't really make much sense to me since it's in-memory