These are chat archives for atomix/atomix
@neverfox thanks. I think I'm probably explaining what we're doing a bit poorly, in the path we're in we don't necessarily try to make a commit to the log, so I'm not sure we get the leadership check we're looking for. Our goal is just to hand out monotonically increasing numbers.
Two approaches we're thinking of:
In (2), we're worried that if the leader loses leadership, it might still hand out new numbers while another leader election takes places and a new leader/ceiling is set. We'd like to get some guarantees around the leader detecting it is no longer the leader. Our non-Atomix-implementation checks for leadership at the beginning of every request (this is relatively cheap compared to setting a new ceiling), but it's not clear (a) how to replicate this behavior with Atomix or (b) if it's necessary.
So I think that begs a couple of questions:
(a) can two nodes believe they're the leader, or do the election cycles guarantee this doesn't occur?
(b) if (a) is true for some overlap, is there a way to ask "is this node still the leader?"
re: sacrificing availability: we'd be comfortable with reasonable short availability loss in the face of a node failure.
Two nodes can certainly believe themselves to be the leader, and because of GC that's basically impossible to prevent. So, the way that case is handled is by using the epoch that's provided in Atomix' leader elections: the
term. While Atomix can't guarantee two nodes won't believe themselves to be the leader at the same time, it can guarantee that no two leaders will ever have the same term and that terms are monotonously increasing. So, when you pair the term with your monotonically increasing number you can ensure consistency. For example, if one client's gets the number
10 from a leader with term
2 and another also gets the number
10 but from a leader with the term
4, you consider the second client to have the consistent value since
4 > 2. Furthermore, since
10 is also a monotonically increasing number that was consistent in term
4, any number greater than
10 with a term less than
4 can be considered inconsistent and that client should request a new number. Because Atomix (or more specifically Raft) guarantees that no two leaders will have the same term and that terms are monotonically increasing, performing this "fencing" ensures that only numbers from the most up-to-date leaders are being used. Often, this type of thing is done with databases that support CAS operations.
It's certainly also possible to check whether a leader is still the leader. That hasn't been added to the API but is simple to add, and indeed checking whether a node is a leader is much more efficient than writing an increment to disk. But it doesn't totally solve the problem, which is why it hasn't been implemented in the Atomix API. It's still theoretically possible for a client to check whether it's still the leader and have e.g. a long GC pause between the time it gets the response and the time the client code reacts to the fact that it's still the leader. So, while not likely, it's still possible for a node to believe itself to be the leader after checking but for it to have lost its leadership. For that reason, either using
DistributedLong or using the term as a fencing token is the only truly safe solution.