These are chat archives for atomix/atomix

28th
Mar 2018
rbondar
@rbondar
Mar 28 2018 03:02
@kuujo dropped email with example
terrytan
@txm119161336_twitter
Mar 28 2018 03:27
hi ,Jordan , i find you made change with member joining, adding promotable state, for copycat1.x, if an active node tries to join, leader received it's request, it will config locally first, then sends to followers , we assume only one follower got config and finished config, after that, leader is down, then the newly joining server will not receive config,he has no idea about other followers ,but the follower finished config will take that joiner into account, this will cause bug ,like the follower will never get votes from majority, that is the reason why you made this change on copycat 2?
Jordan Halterman
@kuujo
Mar 28 2018 04:59

@txm119161336_twitter the reason for the PROMOTABLE role is actually to allow the leader to catch up a follower before it becomes a voting member. When a node is added to the Raft cluster, it’s first added as a PROMOTABLE member which means that it can’t vote in elections but does receive entries from the leader. Once it’s caught up to the leader, it’s then promoted to an ACTIVE voting member.

The reason this is done is because adding a node that doesn’t have all committed entries can block writes to the cluster. For example, if a 2 node cluster adds a third node and then one of the original 2 nodes crashes, the cluster becomes blocked until the leader can replicate all its changes to the added node. By catching it up before it becomes a voting member we reduce the chance the cluster can become unavailable.

terrytan
@txm119161336_twitter
Mar 28 2018 05:27
for the above case i mentioned , is it a problem when it is in copycat1.x? I have done such kind of test ,but for the coding logic, it will be like this ,right?
because the joiner server will blocking the server which is not in it's memberlist , while the follower finished config will see the joiner as a voting member without the promotable state
Jordan Halterman
@kuujo
Mar 28 2018 07:35

Yeah, that’s not the reason for this particular change, but TBH I’m not confident in the Raft configuration protocol for exactly this reason. The protocol is not proven, and configurations are inevitably applied at different times on different nodes.

The idea behind the Raft configuration change protocol is:
• The leader logs a configuration change entry and immediately applies the change
• Nodes apply the change as the configuration change entry is replicated

But there’s then a chance that a configuration change may be applied to one or more nodes but never actually be committed. So, I think to handle this safely, nodes would actually have to preserve a record of the prior configuration to be able to revert to that configuration in the event a configuration change is unsuccessful (some other term is committed at the configuration change index). Nothing like this is implemented in either version.

Another option would be for the leader to commit the change without applying it and then count the number of nodes that have applied it to determine when it’s safe to commit the next configuration change. This seems to me like a safer albeit more complicated approach. But at least it guarantees a configuration change cannot be undone after it’s applied.

Anyways, I’m another note, I introduce support for member groups (zone/rack/host aware replication): atomix/atomix#463

I need to do some more experimenting with the implementation to gain some confidence it’s correct, but it sure looks cool :-P

on*
terrytan
@txm119161336_twitter
Mar 28 2018 07:42
if we have got promotable phase, then i think at least it is safe ,ether the follower having new change is elected to be leader(it will sync it's config to all) ,or the one without new config wins (simply the new config will be lost), but at least it is safe.
Jordan Halterman
@kuujo
Mar 28 2018 18:44
Well, there are actually two configuration changes that take place now - one to add a PROMOTABLE member, and one to promote it to ACTIVE. The latter (which is the important one) still suffers from the same limitations. I actually need to bring this up on the Raft forum since nobody seems to question the orthodoxy wrt configuration changes in Raft.
Need to re-read the relevant portions of the Raft dissertation first though
Johno Crawford
@johnou
Mar 28 2018 19:53
@rbondar not just clusters, nodes