These are chat archives for atomix/atomix

31st
Mar 2016
Richard Pijnenburg
@electrical
Mar 31 2016 08:18
Ha i wish. someone decided to jump infront of a train causing delays up to 1,5 hr and one station where i live they didn’t stop for a while. so had to go to an other station and take a cab
Jordan Halterman
@kuujo
Mar 31 2016 16:39
Wow
Richard Pijnenburg
@electrical
Mar 31 2016 17:09
I know you are very busy with the release stuff. but if you have some time for those examples would be appriciated ;-) but no hurry
Trying to change the world at my job anyway lol
Jordan Halterman
@kuujo
Mar 31 2016 18:58
Yeah I was about to do them last night and then fell asleep haha
Might be after I finish all the docs and the website which needs to be done by Monday
Then I have to put together a three part talk on Atomix, Raft, and Copycat for my work :-( haha
Never ending
But it's really quick to throw this together
Richard Pijnenburg
@electrical
Mar 31 2016 19:00
Haha fair enough ;-)
no hurry for this stuff. unless you have the time to do it quickly ;-)
Richard Pijnenburg
@electrical
Mar 31 2016 19:06
think i just need a push in the right direction to understand it. this is the most difficult project i started so far i think :p
Jordan Halterman
@kuujo
Mar 31 2016 19:06
Yeah I get it
No worries I've made a bunch of stuff like this
Richard Pijnenburg
@electrical
Mar 31 2016 19:07
my lack of java knowledge doesn’t help either. haha
would love to push something working in their face :p and show how things can be if you put some effort in it :-)
Haha that sounds fun :-)
Richard Pijnenburg
@electrical
Mar 31 2016 19:11
vertigo sounds a lot of the base i;m trying to build ?
or a i reading things incorrect?
Richard Pijnenburg
@electrical
Mar 31 2016 19:21
i most likely am :p lol
Jordan Halterman
@kuujo
Mar 31 2016 19:22
No it is a lot like it for sure
Richard Pijnenburg
@electrical
Mar 31 2016 19:23
hehe okay. I’m not to drunk then :p
Jordan Halterman
@kuujo
Mar 31 2016 19:23
Atomix was originally something I was developing for Vertigo, but the only reason I stopped working on Vertigo is because Atomix/Copycat are a full time second job already
Richard Pijnenburg
@electrical
Mar 31 2016 19:23
ahh okay :-)
Jordan Halterman
@kuujo
Mar 31 2016 19:24
But the concepts are very similar. Components have inputs and outputs, and the connections between components are abstracted
Richard Pijnenburg
@electrical
Mar 31 2016 19:24
i see
guess you can give me a lot of good idea’s then i can implement in my project for this.
all the other ‘magic’ logic i have to build
like distributing the pipeline and parsing the config
think i’ll have 2 code paths. the master one which does the controlling. and the node one which builds up the pipeline on a processing node
the master part has to tell the nodes what they need to build up
Jordan Halterman
@kuujo
Mar 31 2016 19:30
Vertigo is distributed but not fault tolerant, which is where Atomix comes in. Now it's easy to do the fault tolerant part. The hard part about adapting LS to that is probably making the inputs and outputs understand distributed concepts. It's easy to do the filtering because those seem to be stateless. But to make it linearly scalable, multiple inputs and outputs also need to be distributed. That means if there are multiple instances of a pipeline, each instance needs to know what data it is supposed to read. They can't all read all the data because that's not scalable. For example, if you were reading from Kafka, each input could each read from a different partition, and all the inputs together read from a topic. The I/O is generally what needs to be scaled because filtering is cheap
Richard Pijnenburg
@electrical
Mar 31 2016 19:32
yeah indeed. but as long as i build an input in a way that can run in multiple threads. then distributing it across nodes is easy
things that receive data is very easy. like a TCP input
Jordan Halterman
@kuujo
Mar 31 2016 19:33
True
Richard Pijnenburg
@electrical
Mar 31 2016 19:33
I’m not super worried about that part.
I’m very worried about the core part to make it as dynamic as possible
back in 5. need to catch my train
Jordan Halterman
@kuujo
Mar 31 2016 19:35
If you can scale an input across il tile threads without synchronization between threads then making it distributed is easy. But often the synchronization part is what hurts distributed systems
multiple*
Richard Pijnenburg
@electrical
Mar 31 2016 19:40
back
yeah very true. for example redis input reads in batches but locks that batch in a way which solves that problem :-)
Richard Pijnenburg
@electrical
Mar 31 2016 20:22
finally home
Jordan Halterman
@kuujo
Mar 31 2016 20:49
:house:
Richard Pijnenburg
@electrical
Mar 31 2016 21:02
I do wonder how i’m going to solve it with plugins that only should run once. the lock resource should help with that
Jordan Halterman
@kuujo
Mar 31 2016 21:02
indeed
Richard Pijnenburg
@electrical
Mar 31 2016 21:03
and will have to use the distributed set or what ever for state saving in case it moves
Jordan Halterman
@kuujo
Mar 31 2016 21:05
the general way this should be done is create a DGroup, elect a leader, and have the leader tell all the members what to do:
DistributedGroup group = atomix.getGroup(“stuffs”).get();
LocalMember localMember = group.join().get();
group.election().onElection(term -> {
  if (term.leader().equals(localMember)) {
    // do stuff
    group.members().forEach(member -> {
      member.tasks().submit(“do it!”).thenRun(() -> {
        System.out.println(member.id() + “ done doing it!”);
      });
    });
  }
});
indeed
actually I should change that API a bit
localMember.election().onElection(term -> {
  group.members().forEach(member -> {
    member.tasks().submit(“do it!”);
  });
});
Richard Pijnenburg
@electrical
Mar 31 2016 21:06
I was thinking of the master process saving a config in the distributed set per node
Jordan Halterman
@kuujo
Mar 31 2016 21:06
electing a single leader to coordinate the cluster is much more efficient than locking if possible
since a leader only needs to be elected once, and a lock usually needs to be acquired on each state change
Richard Pijnenburg
@electrical
Mar 31 2016 21:07
well. in this case the plugin won’t run on the master.
the masters should run nothing except management
Jordan Halterman
@kuujo
Mar 31 2016 21:08
But the leader is telling the other nodes what to do, rather than the other nodes coordinating with each other to decide what to do
Richard Pijnenburg
@electrical
Mar 31 2016 21:08
I see.
but i need to submit the config anyway what pipeline(s) they need to run? or can i do that via the same way?
also need access to the node anyway since they have a local config deciding what they can run
which i need to feed back somehow
Jordan Halterman
@kuujo
Mar 31 2016 21:35
Messages
Richard Pijnenburg
@electrical
Mar 31 2016 21:36
how do you mean ?
Jordan Halterman
@kuujo
Mar 31 2016 21:37
member.tasks().producer("config").submit(task) can send a configuration to a member, and member.messages().producer("config").send(message) can send a message to a node asking for its configuration.
Richard Pijnenburg
@electrical
Mar 31 2016 21:39
ahh okay. interesting
Jordan Halterman
@kuujo
Mar 31 2016 21:42
Leaders should make all decisions and should tell all other nodes what they should be doing. Coordinating a cluster with a bunch of decision makers is a lot more complicated and way more difficult to make fault tolerant. If the leader is coordinating the cluster, only the leader needs to know the status of all other nodes, and the leader can unilaterally react to failures (e.g. tell some other node to take over). The leader can ask other nodes for input (their configuration) but only the leader should decide what that means.
Could also just store node configurations in the cluster too
Richard Pijnenburg
@electrical
Mar 31 2016 21:45
very good point yeah
Jordan Halterman
@kuujo
Mar 31 2016 21:46
That also allows the leader to be more intelligent. It's hard to coordinate a bunch of decision makers to make intelligent decisions. For example, node A thinks it can run x, but node B does too. They have to fight about it and don't have the context of what's running on all the other nodes. The leader can have that context and pick the best node. For example, the leader can say node A is already running more crap than B so assign it to B
Anyways, I guess that's why in general it's easier just to have all the decisions made in the leader if there's no reason those decisions need to scale.
Richard Pijnenburg
@electrical
Mar 31 2016 21:48
very good points. didn’t think about it in that way :)
Richard Pijnenburg
@electrical
Mar 31 2016 21:55
then the node can be much simpler.. it only needs to start the pipeline based on what the master sends out
Richard Pijnenburg
@electrical
Mar 31 2016 22:04
how would it know what to send to which member though? would be a long loop going through all the members?
Jordan Halterman
@kuujo
Mar 31 2016 22:29
Yeah... The leader just has to make that decision somehow.
@electrical is there versioning in ES? I can't find anything
Jordan Halterman
@kuujo
Mar 31 2016 22:35
nvm :-)
Jordan Halterman
@kuujo
Mar 31 2016 22:42
I suppose it stores a version number but not the versioned datas :-(