These are chat archives for atomix/atomix

28th
Mar 2016
Jordan Halterman
@kuujo
Mar 28 2016 00:08
@madjam I’m going to submit a PR for some relatively naive performance improvements. Would love to get a :+1: from you if you still have the benchmarks around
I have exactly zero time :-)
not even enough time to do the things I am already working on
Richard Pijnenburg
@electrical
Mar 28 2016 00:08
haha lol yeah
Jordan Halterman
@kuujo
Mar 28 2016 00:56
Actually... I may have found a bug in the client when submitting multiple requests at the start of a connection. When I submit synchronous requests it handles the connection fine, but I think multiple concurrent requests when the client is connecting to a new node may force it to drop its connection and try again. The client just takes too many rounds to establish a connection.
Going to test against master to make sure I didn't just break something else
Jordan Halterman
@kuujo
Mar 28 2016 01:06
nvm I just broke something :-P
Richard Pijnenburg
@electrical
Mar 28 2016 01:08
haha
shouldn’t be breaking stuff you know :p lol
Jordan Halterman
@kuujo
Mar 28 2016 01:33
actually I don’t think I broke anything. It is a bug but it’s a confusing one
Richard Pijnenburg
@electrical
Mar 28 2016 01:34
ha okay
Jordan Halterman
@kuujo
Mar 28 2016 01:39
yeah definitely looks like a bug
awesome
Jordan Halterman
@kuujo
Mar 28 2016 01:50
ahh I think I found it :-)
Richard Pijnenburg
@electrical
Mar 28 2016 01:50
nice!
what is it ?
Jordan Halterman
@kuujo
Mar 28 2016 01:51
yeah I fixed it...
That connection is designed to automatically reconnect to another node in the cluster when a connection fails
it’s a bunch of asynchronous code and thus recursion
The logic is pretty crazy, but basically the client will learn about new servers when it sends keep-alives to the cluster. When the client learns that the structure of the cluster changed, it resets its connection so it can connect to a potentially better server in the cluster: https://github.com/atomix/copycat/blob/master/client/src/main/java/io/atomix/copycat/client/util/ClientConnection.java#L163-L168
For example, if the client is configured with the FOLLOWERS selection strategy, it will prefer to connect to and communicate with followers to scale reads
But if there’s no follower to connect to then it will connect to the leader
when the client discovers new nodes it resets its connection to allow those types of strategies to ensure they’re connected to the optimal server
Anyways, the bug occurred in that block I linked above
When a bunch of concurrent writes are submitted and the cluster’s structure changes, a bunch of writes concurrently reset the connection, when it should only be the first write that resets the connection
So, the fix...
Jordan Halterman
@kuujo
Mar 28 2016 01:57
#202
Jordan Halterman
@kuujo
Mar 28 2016 02:10
Other than that, I have made a bunch of optimizations to memory mapped files and added pipelining, but the pipelining right now is significantly slower than batching only. In master, entries are batched really well under high load. But with pipelining, even while only allowing two concurrent requests, each AppendRequest ends up sending only a single entry, and that's far more overhead than batching alone.
Jordan Halterman
@kuujo
Mar 28 2016 02:16
It sends two every now and then, and three even less frequently
but absent pipelining batches are in the dozens under high load
Richard Pijnenburg
@electrical
Mar 28 2016 02:21
can’t find #202
ah its in copycat
Jordan Halterman
@kuujo
Mar 28 2016 02:21
Yeah
Gitter doesn't handle multiple repos wel
Richard Pijnenburg
@electrical
Mar 28 2016 02:22
you can do atomix/copycat#202
Jordan Halterman
@kuujo
Mar 28 2016 02:24
What did you do?
Richard Pijnenburg
@electrical
Mar 28 2016 02:24
type in the org/repo#number
this way you can link things from other repo’s and orgs
Jordan Halterman
@kuujo
Mar 28 2016 02:25
Ahh I think I only tried copycat#202
atomix/copycat#202
Excellent
Richard Pijnenburg
@electrical
Mar 28 2016 02:25
:-)
never to late, or to early in my case to learn/teach something :p
that reminds me, i should go get some sleep lol
almost 3:30 am
Richard Pijnenburg
@electrical
Mar 28 2016 02:46
hmm. is there a way to alias the run method of callable into something else? for example i got the run method in inputs.
argh. better get some sleep and look at it again tomorrow and read some more.. its hard to understand it and trying to see how i could abstract it in a nice usable way
Jordan Halterman
@kuujo
Mar 28 2016 03:04
No there's no way to alias. You just need to learn design patterns. For example, create an adaptor class that implements Runnable to invoke x on the underlying object. Typically, there's no overhead to that type of thing as Java's compiler will basically make it a direct method call
Jordan Halterman
@kuujo
Mar 28 2016 08:48
I made some minor performance improvements to memory mapped log segments. My tests showed them sustaining 40k writes/sec in a three node local cluster on my laptop, and buffers are still flushed to disk prior to responding to clients, and thus all the same guarantees hold as with DISK logs. I also experimented a bit with an alternative approach - buffering writes externally and flushing on my the committed writes on commit - but that proved slower than just flushing MAPPED segments in my tests. Pipelining too proved fruitless. I was unable to get pipelining to outperform straight batches because of the significantly reduced batch sizes in pipelining.
atomix/copycat#203 has the performance improvements for MAPPED logs.
Richard Pijnenburg
@electrical
Mar 28 2016 17:28
@kuujo i have something working now with Runnable and a linkedList as queue to pass data on between things
Richard Pijnenburg
@electrical
Mar 28 2016 17:41
hmm if i have a Executors.newSingleThreadExecutor(); and start a runnable in there which has a while loop. ( for inputs ) it seems to be blocking the entire app?
Richard Pijnenburg
@electrical
Mar 28 2016 18:05
ah. seems its being executed in the main loop