These are chat archives for atomix/atomix
I think it's perfect for Atomix. A lot of people come here asking how to use Atomix to build a database. You came here asking how to coordinate a database. That's what you should be doing.
I think you're doing the right thing here. Atomix internally uses a point to point messaging abstraction with a Netty implementation that will eventually be exposed to users, but isn't well suited to general messaging. Anyways, you should just assign metadata to members when they join, e.g.
group.join(new MyMetadata()) to associate information with a group member.
I think there will be a lot less chaos than you think there will be when node goes down even if you have a thousand groups and a thousand leaders. The reason is from the underlying state machine's perspective, a node crashing is still a single event. It will appear to each group as a bunch of separate events, but a single node crashing is 1000 method calls in the state machine, which is trivial. The only real overhead will be a single commit to the Raft log for the expired session (which triggers leader changes in all the state machines) and servers sending leader change events to clients, but again 1000 leader election events is fairly trivial as they're simply TCP messages.
In the past, there have been utilities for consistent hashing. But they've been removed in favor of simpler abstractions for now. I suppose the way to do consistent hashing would be to use
DistributedGroup for cluster membership/failure detection and hash nodes/keys using the group. This is actually the one major distributed systems algorithms that I don't have a reference implementation for, but have one planned.
Here's what I tell new users exploring ways to use Atomix: Atomix and it's primitives can be powerful if used correctly, but it should be seen merely as a component of a solution and not a solution in and of itself. There are obviously limitations to consensus in terms of performance and fault tolerance. People are often too eager for a total solution, but I think you're thinking about this exactly the right way.
FWIW, in my distributed-stream-processing project, I've implemented consistent hashing in Atomix as follows:
I have a DistributedGroup representing the nodes in the cluster. Each node joins the group at startup, and builds an identical hash-ring from the list of member-ids upon joining. Then each node listens for
onLeave events, and updates its local view of the ring accordingly.
To ensure orderly rebalancing transitions, I also establish a DistributedLock for each resource ("partition") that is assigned via the ring. When a node takes ownership of a partition, it waits to obtain that lock before beginning processing. Relinquishing a partition involves releasing that lock after processing is halted/flushed and checkpoints have been captured.
onJoinis is there any message delivery guarantees for those listeners or like zookeeper one is expected to poll for state changes
onLeaveevents in the same order, but @kuujo would probably be better qualified to confirm that