These are chat archives for atomix/atomix

27th
Mar 2016
Jordan Halterman
@kuujo
Mar 27 2016 00:01 UTC
But there doesn't really seem to be any good solution for this problem, and that's why it hasn't been implemented. This is not effectively any different than what it's like now. If a node is started with a list of known nodes in the cluster, it will join the cluster.
Using any sort of discovery is just a way to get that list dynamically, with the constraint that there must be a list
Richard Pijnenburg
@electrical
Mar 27 2016 00:02 UTC
yeah indeed.
the only big difference between mutlicast and any other discovery is that multicast can do its ‘magic’ to find other nodes.. other discovery systems would define that list in an other way
Madan Jampani
@madjam
Mar 27 2016 00:02 UTC
the first node might find an empty list and start a single node cluster, right?
Jordan Halterman
@kuujo
Mar 27 2016 00:03 UTC
As long as a node is started with the knowledge of at least one other server that is a member of the cluster, it cannot cause a split brain since it cannot unilaterally elect itself leader. This takes advantage of the single member change
Yeah that's why there needs to be something that says whether there should be at least one other node, but that's where it gets sloppy and confusing and sort of unusable
The first node that's started can elect itself leader. That's what Raft recommends. Start a single node cluster, add a server, add a server
As long as all servers started after the first server know about at least one server in the cluster they can't cause a split brain
Richard Pijnenburg
@electrical
Mar 27 2016 00:04 UTC
you could do ‘if im the only node and i don’t have a list of nodes to connect to’ then if my bootstrap flag is true i can elect my self as master. otherwise i should be part of an existing cluster.
Jordan Halterman
@kuujo
Mar 27 2016 00:05 UTC
That's the function the members list serves now
Right
Richard Pijnenburg
@electrical
Mar 27 2016 00:05 UTC
and i forsee this feature mainly to be used in client setup’s where they autoscale the compute part for example
as an example for my logstash project
but i agree protection is needed
Jordan Halterman
@kuujo
Mar 27 2016 00:08 UTC
Yeah it can be done for clients. There's no risk in that. Only servers would need a bootstrap flag saying they can form a new cluster.
Richard Pijnenburg
@electrical
Mar 27 2016 00:08 UTC
yeah indeed
Jordan Halterman
@kuujo
Mar 27 2016 00:09 UTC
It's only needed for election protection. But I don't anticipate this being done any time soon. I may just provide an interface that can be inple
ugh phone
Implemented for discovery, and if someone contributes it back I'll take it
Richard Pijnenburg
@electrical
Mar 27 2016 00:09 UTC
If you could build the interface, i’ll build the discovery stuff :-)
shouldn’t be to hard
Jordan Halterman
@kuujo
Mar 27 2016 00:10 UTC
Cool
Richard Pijnenburg
@electrical
Mar 27 2016 00:10 UTC
EC2 wise i can peak at elasticsearch :p
but first.. sleep :-) past midnight. :-)
was a wake at 7 am :-(
Jordan Halterman
@kuujo
Mar 27 2016 00:17 UTC
The bootstrap flag approach makes sense. Users just like to misconfiguration things and then the project gets blamed for safety issues. A user sets bootstrap = true on all nodes and then complains about split brain. It's not even safe to set it on more than one node. But I've thought about this a lot, and the same is true of the configuration now. Any user can misconfiguration the cluster and cause split brain simply by e.g. starting multiple nodes that aren't members of the cluster but which are both partitioned and including each other in their configurations. Both misconfigurations are user error. But I think the bootstrap flag is a little simpler and allows any type of unreliable discovery to be done
Jordan Halterman
@kuujo
Mar 27 2016 01:17 UTC
interesting
I see two references to bootstrapping in the Consul docs. The first reference seems to be more like what Atpmix does now, and the second "manual bootstrap" seems more like what we're proposing. Either is fine. They seem to have moved away from the latter in favor of the former, but I think it's a bit of a different use case since Atomix is a framework
Anyways, good conversation. Will try to flesh this out some time soon. I want to implement the performance fixes for Copycat and get it released first.
Richard Pijnenburg
@electrical
Mar 27 2016 09:47 UTC
@kuujo cool. Yeah, release the other things first. this is all just extra features that’s just nice to have. not mandatory :-)
Jordan Halterman
@kuujo
Mar 27 2016 18:18 UTC
I’m not sure it’s going to be very easy to implement parallel appends for leaders in Copycat. The problem is, the way Copycat appends and replicates entries sort of precludes it. The LeaderAppender expects entries to be replicated to already be present in the log. One easy way to get around this is to hold the entries-to-be-appended in an in-memory queue and write them on flush. That way, the LeaderAppender can read the entries for replication, but they’re not actually written until after replication has started. The leader just has to keep track of the highest index it flushed to disk.
Jordan Halterman
@kuujo
Mar 27 2016 18:28 UTC
Further optimizations can then be made under high load to queue multiple writes and flush them all concurrently with append requests
Richard Pijnenburg
@electrical
Mar 27 2016 18:30 UTC
parallel appends? so batching up the commands you mean?
Jordan Halterman
@kuujo
Mar 27 2016 18:31 UTC
no Raft/Copycat already does batching
the performance enhancement is replicating new entries in parallel with appending on the leader
normally, the leader appends an entry and then replicates it
but in this case, the leader first begins replicating the new entry and appends it during replication, thus doing both concurrently
reducing latency
This can be done in Copycat by simply queueing writes to the log, starting the replication, and then flushing the log
Under high load, multiple writes can be queued and replicated in batch
Richard Pijnenburg
@electrical
Mar 27 2016 18:33 UTC
ah i see. so async ? if i understand it correctly when you do a command the master processes it, replicates it and then replies back with an ack right?
Jordan Halterman
@kuujo
Mar 27 2016 18:33 UTC
yeah
Richard Pijnenburg
@electrical
Mar 27 2016 18:33 UTC
so with async it sends the ack when the master processes it
and does the replication in the background
you could have it configurable cluster.replication: sync / async cluster.replication.async.batch_size: x
Jordan Halterman
@kuujo
Mar 27 2016 18:35 UTC
Raft still requires that an entry be persisted on a majority of the cluster before responding to a write. Normally, the leader receives a request, writes it to disk, and then replicates it to a majority of the cluster. This is just changing the order to first replicate and then write to disk while other nodes are also writing to disk. The normal algorithm requires two synchronous writes on the path, and this change just allows leaders to do their write at the same logical time as followers
Richard Pijnenburg
@electrical
Mar 27 2016 18:36 UTC
Ahh okay, i see
write - replicate - write - ack would be the normal process ?
Jordan Halterman
@kuujo
Mar 27 2016 18:40 UTC
I’m also working on pipelining, but there are some challenges to optimizing it. Pipelining would allow multiple in-flight AppendRequests to followers, but the challenge is making sure the leader still batches writes as well. If leaders just arbitrarily pipeline writes then every AppendRequest will be for a single entry and it will probably behave worse than batching without pipelining
But leader’s also can’t wait arbitrarily long for a batch to accumulate
Richard Pijnenburg
@electrical
Mar 27 2016 18:41 UTC
hmm yeah
got an other question, unrelated to atomix but since you know so much i thought i might as well ask you :p
Jordan Halterman
@kuujo
Mar 27 2016 18:42 UTC
alrighty
Richard Pijnenburg
@electrical
Mar 27 2016 18:43 UTC
I’m searching for something to implement a pipeline for that logstash project. Each plugin will have some standard function calls per type of plugin. I’m looking for something to connect them together into a pipeline
With LS they compile the plugins into a single function and thread that i believe.
Jordan Halterman
@kuujo
Mar 27 2016 18:45 UTC
hmm
I don’t have any great ideas
Richard Pijnenburg
@electrical
Mar 27 2016 18:52 UTC
Oh yeah and you gave me pointers to solve my threading problem. Can’t find back in my history anymore what it was called
Jordan Halterman
@kuujo
Mar 27 2016 18:52 UTC
The Executor framework. Should find lots of stuff on that
Richard Pijnenburg
@electrical
Mar 27 2016 18:53 UTC
Ah that was it yeah :-)
Jordan Halterman
@kuujo
Mar 27 2016 18:53 UTC
I don't know about pipelining. It can be done with executors. a call b calls c
Richard Pijnenburg
@electrical
Mar 27 2016 18:55 UTC
okay. I’ll most likely have to build something my self to make it easy :-)
Jordan Halterman
@kuujo
Mar 27 2016 18:59 UTC
Hmm... I think Copycat is going to need some way to understand the request rate to decide whether to batch entries for pipelining. My idea is to use a ring buffer. Timestamps are already stored in entries. The timestamp can be stored in a ring buffer, and if the rate (determined by timestamps in the ring buffer) is greater than n per second, batch writes. When a request is received, store the timestamp in the ring buffer of n size. If the next timestamp is < one second old, buffer the write. Once the buffer is full, flush and replicate. If the rate returns to > n per second, return to single writes.
Richard Pijnenburg
@electrical
Mar 27 2016 19:11 UTC
A pipeline is in its simplist form an ArrayList right? so if i compile a list, for each stage i can initialize an Executor for that part.
Jordan Halterman
@kuujo
Mar 27 2016 19:12 UTC
Of course… it feels like the same thing is affectively acheived using memory mapped files and flushing to disk after starting replication
indeed...
or you can create a set of objects and have each of them call the next
Richard Pijnenburg
@electrical
Mar 27 2016 19:15 UTC
How do you mean? a set of objects?
Every input plugin for example will have an init, run and emit function.. init should only be called once. independent of how many threads i start, run should be done per thread. emit submits it to the next stage
Jordan Halterman
@kuujo
Mar 27 2016 19:21 UTC
makes sense…
There are two ways to go about it. Either there can be a controller that knows a then b then c, or each indivitual stage can have a reference to the next stage.
in the latter case, emit would call the next stage with the input arguments
in the former case, emit would notify the controller that the stage is done, and the controller would call the next stage
Richard Pijnenburg
@electrical
Mar 27 2016 19:22 UTC
Okay. I think the controller way would make more sense. eventually i want to do also things like parallel execution. for example if i have 3 filters and neither of the 3 are dependent on each other. i can run them in parallel for each event
Jordan Halterman
@kuujo
Mar 27 2016 19:22 UTC
indeed
Richard Pijnenburg
@electrical
Mar 27 2016 19:23 UTC
only downside of that is that i need to somehow merge the data from all 3 filters :-)
Jordan Halterman
@kuujo
Mar 27 2016 19:40 UTC
yeah it can get crazy with threading… that’s why so much time was spent on creating a consistent threading model in Copycat
Richard Pijnenburg
@electrical
Mar 27 2016 19:40 UTC
i can imagine
but that’s for the future. for now a single line is fine :-)
one thing im not sure about is how to handle the filter threading. should i compile them as a single ‘thing’ and thread that block. or thread them all individually.
Jordan Halterman
@kuujo
Mar 27 2016 19:44 UTC
probably individually
Richard Pijnenburg
@electrical
Mar 27 2016 19:51 UTC
hmm. each pipeline controller will need to be able to setup the plugin in 2 stages i guess. first is initialize the plugin it self. then thread the run action. and also need some kind of shutdown function to halt the plugin when its an input for example
big question then is, can i share that instantiation. from all the examples i found so far about the executorService is that it uses Runnable right ? and from what i found so far it instantiates the class on each run ?
Richard Pijnenburg
@electrical
Mar 27 2016 20:06 UTC
sorry for all the questions :-)
http://tutorials.jenkov.com/java-util-concurrent/executorservice.html i’m trying to follow these examples to understand it a bit more.. but if i put it inside a class it won’t even compile with javac :-(
ah, found it. didn’t have it in a main function :-)
Jordan Halterman
@kuujo
Mar 27 2016 20:30 UTC
It takes a Runnable, but that doesn't mean that Runnable has to be instantiated each time. It's an instance that you pass. You can instantiate it whenever you want
Well your problem is that you're using javac :-P
Richard Pijnenburg
@electrical
Mar 27 2016 20:32 UTC
haha
Jordan Halterman
@kuujo
Mar 27 2016 20:32 UTC
Use IntelliJ or Eclipse! No questions about whether something will compile. It will tell you. That's a lot better than trial and error. I don't know anyone that uses javac.
Richard Pijnenburg
@electrical
Mar 27 2016 20:33 UTC
hehe yeah. my fault for runnign it on commandline :p
Jordan Halterman
@kuujo
Mar 27 2016 20:33 UTC
Gotta use Maven at least, and use an IDE until you learn the language
Richard Pijnenburg
@electrical
Mar 27 2016 20:33 UTC
habits die hard :p
i thought i installed IntelliJ a while ago
oh no eclipse
Jordan Halterman
@kuujo
Mar 27 2016 20:34 UTC
It's not compiling because the code isn't in a method
Richard Pijnenburg
@electrical
Mar 27 2016 20:35 UTC
yeah. modified it to have the main method and works
starting eclipse now
just a habit of using cli
Jordan Halterman
@kuujo
Mar 27 2016 20:38 UTC
When you pass new Runnable you're passing a new anonymous instance of Runnable. But you can define another class that implements Runnble, create a single instance, and execute it as many times as you want
new Runnable() {...} is just Java 7's form of closures
Richard Pijnenburg
@electrical
Mar 27 2016 20:40 UTC

Hmm okay. But when i tried that i can’t do something like
Runnable myplugin = new MyPlugin();
myplugin.init();

i had to create a method with the same name as the class and then pass it to the new MyPlugin() method ?

Jordan Halterman
@kuujo
Mar 27 2016 20:40 UTC
In Java 8 (which Copycat/Atomix requires) you can do the shorter form too executorService.execute(() -> {...})
MyPlugin just has to implement Runnable. Runnable is an interface that says when MyPlugin implements it, MyPlugin must implement these methods (e.g. run())
Richard Pijnenburg
@electrical
Mar 27 2016 20:41 UTC
I want this project to depend on Java 8 yeah
Jordan Halterman
@kuujo
Mar 27 2016 20:41 UTC
Eclipse or InelliJ will generate the methods for you
In IntelliJ, I implement an interface and type command-i and it generates the interface methods in the class
This is why using an IDE in Java or something capable of this can be a lot faster
No need to look up what methods you're supposed to implement, and no need to even type them
Richard Pijnenburg
@electrical
Mar 27 2016 20:44 UTC
hehe okay, i see
Jordan Halterman
@kuujo
Mar 27 2016 20:45 UTC
Just try to get used to the shortcuts and you can writes codes as fast as any other language
Richard Pijnenburg
@electrical
Mar 27 2016 20:45 UTC
I will try :-)
Im so used to do everything from Vim :-)
Richard Pijnenburg
@electrical
Mar 27 2016 20:51 UTC
okay.. doing a small POC with callable. because i will need to be able to pass on data back and forth.
now trying to find out how i can pass data from run 1 to run 2
Jordan Halterman
@kuujo
Mar 27 2016 21:07 UTC
You don't necessarily have to use a Callable. The other way to pass data between threads without having to block is with multiple Executors or CompletableFuture or something. If you just use Callable and block on a Future to get the result you sort of defeat the purpose of threads. Nothing in Atomix/Copycat ever blocks any thread (except I/O), but all sorts of information is passed between threads. But that came from a lot of experience with asynchronous programming
Richard Pijnenburg
@electrical
Mar 27 2016 21:08 UTC
Ahh okay.
more to read then ;-)
Richard Pijnenburg
@electrical
Mar 27 2016 21:18 UTC
hmm. reading up on java streams which is in java 8
that potentially could be my pipeline together with CompletableFuture
Richard Pijnenburg
@electrical
Mar 27 2016 21:23 UTC
uhg. my head hurts of all this stuff trying to understand it.
Richard Pijnenburg
@electrical
Mar 27 2016 22:58 UTC
Wish there was someone who had some time and actually understands this stuff :p