These are chat archives for atomix/atomix

7th
Oct 2016
Jon Hall
@jhall11
Oct 07 2016 17:21
From the copycat architecture documentation:
However, in order to prevent “split brain”, Raft only allows a single member to be added to or removed from the cluster at any given time, and no two configuration changes may overlap in commitment.
What does that last phrase mean? That each change in configuration is linearized?
Jordan Halterman
@kuujo
Oct 07 2016 18:19

Yep... There are two ways to do configuration changes in Raft, but the original method is no longer recommended. Copycat uses the new method - one server at a time. Yes, only one configuration change can take place at any given time, but Copycat enforces that constraint internally. The rationale for single-member configuration changes is if too many nodes are added at once that can cause a split brain since all nodes cannot be updated at the exact same time. So, one node may see a five node cluster and another a two node cluster. Adding a site node at a time and ensuring changes don't overlap prevent a case where split brain occurs. So, when a configuration change is done, the change is replicated from the leader to all followers. When a follower receives a change, it immediately updates its configuration without even waiting for commitment. Once the leader sees that a change has been replicated to a majority of the cluster the next change can be started. If the leader were to replicate multiple changes simultaneously, multiple nodes could still be added to the configuration of one follower reintroducing the potential for split brain where it could vote for a leader from the new configuration which not all nodes know about. That's impossible with single member changes since the new configuration will never have enough unknown nodes to get a leader elected.

Copycat implements that safety mechanism internally. When a node changes the configuration, that change is forwarded to the leader along with the node's commitIndex. If the leader's last configuration index is greater than the follower's commitIndex for the change, it rejects the change. If another change is already underway, it tells the follower to try again later. The follower will retry until successful or until the leader indicates that the follower's configuration is inconsistent with the cluster's configuration, in which case the user has to resolve the conflict. This is to ensure the user is not making a change to a configuration that has already been changed by another node.

Jon Hall
@jhall11
Oct 07 2016 18:24
Ah, makes sense!