These are chat archives for atomix/atomix

29th
Jun 2017
Jordan Halterman
@kuujo
Jun 29 2017 00:04
@txm119161336_twitter LINEARIZABLE requires a heartbeat after applying the query to verify that the leader is still the leader. LINEARIZABLE_LEASE skips the heartbeat and assumes if the leader hasn't transitioned to another state then it's still the leader. This risks a stale read in in rare cases (between the time a new leader is elected and the time the old leader steps down), but reduces latency by eliminating round trips for reads.
Jordan Halterman
@kuujo
Jun 29 2017 00:13

@aaudiber Copycat will no longer be officially maintained once we transition ONOS to the Atomix Raft implementation. I only get paid to maintain one Raft implementation 😜 Users will have to move to atomix-raft which is just a way better version of Copycat anyways!

That said, the consolidation of Atomix's utilities, messaging, protocols, and algorithms in the Atomix proper repo doesn't change the dependency hierarchy. atomix-raft is still a standalone Raft client/server. It just lives in the Atomix repo now. Moving it into the Atomix repo made sense since it will also house a variety of other distributed systems algorithms like gossip/anti-entropy, failure detection, etc. Below Atomix 2.0 is a library of distributed systems algorithms and utilities that can each be used by themselves. Atomix 2.0 just pieces them together to do all the things I mentioned: partitioning, failure detection, cluster management, messaging, primitives, etc. Those libraries like Raft/gossip/failure detection will be documented and will always be usable as standalone libraries because that's the way we intend to use them in ONOS. Maybe they'll be separated into their own repos again some time in the future, but for now having them all in the Atomix repo has eased the pain of refactoring.

Andrew Audibert
@aaudiber
Jun 29 2017 00:35
Thanks for the detailed explanation, glad to hear you'll still be maintaining atomix-raft as a standalone Raft client/server. Looking forward to using this new and improved version of Copycat :)
Jordan Halterman
@kuujo
Jun 29 2017 01:03
Also FYI I'm actually almost done updating ONOS to use atomix-raft standalone so I know it works :-)
terrytan
@txm119161336_twitter
Jun 29 2017 02:12
Jordan ,if as what you said, LINEARIZABLE_LEASE scenario, if a client submits a command request to leader ,somehow network partition happens , another server takes leadership ,the client will never get the response ,because maybe the old leader will receive a response from another server who's term is greater than the old leader ,then it steps down.
Jordan Halterman
@kuujo
Jun 29 2017 03:50
@txm119161336_twitter it may be possible for the leader not to send a response if it steps down between receiving the request and sending a response. We use timeouts in Netty messaging and retry queries externally. TBH maybe we should make sure the leader still responds to queries after a transition
terrytan
@txm119161336_twitter
Jun 29 2017 05:58
@kuujo Thank you for reply Jordan ,for the keepalive machenism , if the session is expired ,the server will not take responsibility for closing the connection ,the client will do it by the state callback , i
have found the answer myself ,previously ,i raised a question on google forum
Jordan Halterman
@kuujo
Jun 29 2017 11:28

Atomix 2.0 no longer has connections. We found the management of connections in Atomix to be a point of instability. Often clients switch servers too frequently during high throughout, and the overhead of setting up connections exacerbates the issue.

We also found connections between clients and followers to be unexpectedly expensive when events are frequently sent to clients. In ONOS, often every command will trigger an event. The problem is, if the client is connected to a follower, the command has to be replicated to that follower and committed to trigger the event to the client before the client can handle any later responses. So, of the client submits two commands to a follower and he first command triggers an event, often the client becomes blocked waiting for the event before it can complete the second command.

So, to mitigate these two problems, Atomix 2.0 uses peer-to-peer messaging (no connections), and events are always sent by the leader. That means connections are never created or closed - at least at the Raft level - and clients' responses are rarely blocked waiting for events. Perhaps this places a bit more load on the leader, but partitioning in Atomix 2.0 will scale leaders anyways.