These are chat archives for atomix/atomix

10th
May 2016
Madan Jampani
@madjam
May 10 2016 07:20
Encountered some weird behavior when submitting queries. System would stall all of a sudden and there would be a leadership change. After some code digging I may be uncovered a “minor” issue with the pipelining logic
It took a while to accurately line up the sequence of steps but I think I have a fair understanding of what was going on (or not going on :))
But the gist of it is MemberState::canAppend should never return false if appending == 0
Madan Jampani
@madjam
May 10 2016 07:38
The longer form of the issue: A LINEARIZABLE query needs to perform a quick heartbeat to ascertain quorum. But the corresponding AppendRequest cannot be sent out because MemberState::canAppend returns false. This leaves heartbeatFuture as non-null and subsequent heartbeat’s get queued up. Only way to exit this impasse if is if a new entry is appended to the log (A keep-alive will do) and replicated. But if the followers timeout on leader heartbeat before that happens, they will start a new poll and one of them will take over leadership.
Jason Savlov
@jsavlov
May 10 2016 15:49
Is it possible for an AtomixClient to modify data on a replica using distrValue.set()?
Jordan Halterman
@kuujo
May 10 2016 16:35
@madjam interesting... I'm going to poke around and see if I can understand what you're saying, but sounds like a good find
@jsavlov yes. AtomixClient and AtomixReplica and the standalone server are all completely interchangeable. A cluster just has to have at least one replica or server, and clients can operate on that state. A replica is literally just a client/server combined. When you're working with a replica you're actually just using a client that passes requests directly to the underlying server. To modify the state of a replica from a client, just create a DistributedValue and do you're thing. All instances of the DistributedValue with the same name point to the same state in the cluster. So, DistributedValue "foo" created on a replica is the same distributed object as DistributedValue "foo" created on a client. They're both backed by the same replicated state machine.
Jordan Halterman
@kuujo
May 10 2016 16:44
@madjam so adding appending == 0 || to MemberState::canAppend should fix it?
Jason Savlov
@jsavlov
May 10 2016 17:04
@kuujo That's what I think I'm doing.. for some reason, it won't execute the thenAccept() part of my getValue() call... it just skips right over
Jordan Halterman
@kuujo
May 10 2016 17:06
So, it's possible the future is being completed with an exception. In that case, callbacks that assume non-exceptional completion will not be called, e.g. thenAccept or thenRun. If you use whenComplete it will probably be called with a non-null error perhaps due to serialization or something like that.
Jason Savlov
@jsavlov
May 10 2016 17:15
Nope, it's not even giving me a Throwable.. The debugger skips right over the code
Madan Jampani
@madjam
May 10 2016 17:18
@kuujo that is precisely what I did in the PR request
Jason Savlov
@jsavlov
May 10 2016 17:31
So I asked my professor about it (Doug Lea is my professor.. talk about useful resources for concurrency issues). Turns out the lamba expression blocks, but because it's asynchronous it just keeps going