These are chat archives for atomix/atomix

3rd
Apr 2016
Jordan Halterman
@kuujo
Apr 03 2016 01:08
indeed
Jonathan Halterman
@jhalterman
Apr 03 2016 01:13
Ooh did I miss out on more naming fun?
Jordan Halterman
@kuujo
Apr 03 2016 01:13
indeed
we decided on SchoolBusControllerServiceConfigFactoryAdapter
Adaptor
Jonathan Halterman
@jhalterman
Apr 03 2016 01:15
you know i'm happy as long as SchoolBus is in there
Richard Pijnenburg
@electrical
Apr 03 2016 01:15
Whahaha
Jonathan Halterman
@jhalterman
Apr 03 2016 01:16
@kuujo how comes the messaging api?
@electrical normally @kuujo and i debate naming via IM so you guys miss all that fun. maybe we should do it here in chat
...next time
Jordan Halterman
@kuujo
Apr 03 2016 01:17
@With(Children.class)
@Because(“school is out”)
@Uses(Parent.class)
class SchoolBusControllerServiceConfigFactoryAdapter implements SchoolBus, Controller, Service, Config, Factory, Adapter, AnnotationLover {
  ...
}
haha yeah
Jonathan Halterman
@jhalterman
Apr 03 2016 01:17
LOL
those annotations - gotta love it
Jordan Halterman
@kuujo
Apr 03 2016 01:17
haha the markdown highlighted AnnotationLover
guess it knows what’s up
Jonathan Halterman
@jhalterman
Apr 03 2016 01:18
it's like some magic keyword
Jordan Halterman
@kuujo
Apr 03 2016 01:18
I’m just finishing up the last part of it - request-reply
they must love annotations at Gitter too
Jonathan Halterman
@jhalterman
Apr 03 2016 01:18
so it'll do single consumer and broadcast?
Jordan Halterman
@kuujo
Apr 03 2016 01:18
indeed
Jonathan Halterman
@jhalterman
Apr 03 2016 01:19
via publisher config?
Jordan Halterman
@kuujo
Apr 03 2016 01:19
indeed
Jonathan Halterman
@jhalterman
Apr 03 2016 01:19
i am going to stand by my consumer group opinion for now and wait till you come around - maybe a month from now :)
i'm fine with the less is more argument, but eventually i don't see a problem with some basic messaging stuff like that
Jordan Halterman
@kuujo
Apr 03 2016 01:20
will just have to be revisited later
the direct messaging too
Jonathan Halterman
@jhalterman
Apr 03 2016 01:20
could be interesting to revisit in conjunction with partitioning
along with many other things
we do have a messaging module user though
at least 1 :)
i'll let you break the bad news to him/her
Richard Pijnenburg
@electrical
Apr 03 2016 01:54
@kuujo i did read something about CompletableFuture’s in the context of threading. is there a big difference between those and executor service?
Jordan Halterman
@kuujo
Apr 03 2016 02:12
CompletableFuture uses Executors for threading
That is, CompletableFuture is sort of an improvement to the Executor framework for asynchronous promises
Richard Pijnenburg
@electrical
Apr 03 2016 02:14
‘sort of’ doesn’t sound all to convincing :p
if i remember you use it in atomix right?
Jordan Halterman
@kuujo
Apr 03 2016 02:15
Well sort of in that they're not the same thing. It's sort of an improvement on the Executor framework in that futures use that framework, but it's also a separate API that is just tackling a different problem
Well... I guess by sort of I mean it's not necessarily just an improvement on the Executor framework, it's a legitimate separate API
CompletableFuture uses Executors, but Executor is just an interface. The executor framework is the implementation
Richard Pijnenburg
@electrical
Apr 03 2016 02:17
hehe okay. for my project would you adivse to stick with the executor framework or completeablefuture stuff?
Jordan Halterman
@kuujo
Apr 03 2016 02:17
Copycat and Atomix use CompletableFuture extensively and it's great but has a bit of a learning curve
But it's 100x better than normal Future
Richard Pijnenburg
@electrical
Apr 03 2016 02:18
hehe okay :-)
Jordan Halterman
@kuujo
Apr 03 2016 02:18
CompletableFuture and the Executor framework are what allow Copycat to handle tons of concurrent requests in a single thread
Using Executor as an event loop and CompletableFuture for asynchronous callbacks
And I would definitely recommend it
Richard Pijnenburg
@electrical
Apr 03 2016 02:20
okay. I’ve been reading up on it. still trying to wrap my head around it all.
Jordan Halterman
@kuujo
Apr 03 2016 02:20
@jhalterman the user you're talking about is likely not losing functionality
Jonathan Halterman
@jhalterman
Apr 03 2016 02:20
i just saw a mention of messaging module
true that tho
Richard Pijnenburg
@electrical
Apr 03 2016 02:22
@kuujo if you find some time to make an example or something how i could use it for building the pipeline would be great. reading most of the day about it all, just hard to get a real grasp of it.
Jonathan Halterman
@jhalterman
Apr 03 2016 02:23
@electrical did you check out this writeup? http://www.nurkiewicz.com/2013/05/java-8-definitive-guide-to.html
Richard Pijnenburg
@electrical
Apr 03 2016 02:24
@jhalterman yeah. read that one several times trying to understand it
Jonathan Halterman
@jhalterman
Apr 03 2016 02:24
the CompletableFuture methods all have different names, but mostly do the same thing - allow you to take some input and spit out an output
apply, accept, run, handle - all similar
Richard Pijnenburg
@electrical
Apr 03 2016 02:27
for the filters that’s mostly the case indeed, have an input, do something with that and send it further.
inputs are quite different since they have to run in their own thread and read from source x
which is usually a remote thing
Jordan Halterman
@kuujo
Apr 03 2016 02:28
You only need CompletableFuture if you need something like a request and response across threads. That's what it's for.
Richard Pijnenburg
@electrical
Apr 03 2016 02:28
ah okay. i think my threads are fairly isolated, except that they need to do something with the data
Jordan Halterman
@kuujo
Apr 03 2016 02:29
Call a method that returns a CompletableFuture, and that method completed the future when it's done doing what it's supposed to do. When the future is completed, the future's callbacks are called
electrical @electrical is confused :-)
Jordan Halterman
@kuujo
Apr 03 2016 02:32
I'm still not sure Atomix should ever have anything that's not... atomic. I think it should do one thing and do it well. When we start adding the plethora of messaging and partitioning and replication strategies to Atomix, it will take another year or two to make those legitimately useful and competitive with frameworks that focus on those things, and it will take away from improving what Atomix is good at and should be good at: atomic operations in a distributed system. If Atomix were a replication framework it would make sense, but it's not. I just don't want something to be half done in Atomix. If Atomix is going to be a messaging and replication framework then there is a ton of refactoring that needs to happen to make it practical for large scale distributed systems. Raft is not designed for general messaging, and it shouldn't be used that way. Unless/until the right abstractions are found to provide real value for building different types of opinionated replication algorithms (other than Raft), I don't think anything related to ha sling of actual data in replication should go into Atomix. It is a coordination framework, and a replication framework should likely be elsewhere.
Asynchronous programming is difficult to understand if you haven't done it before. It's just a completely different way of thinking about the flow of programs
Richard Pijnenburg
@electrical
Apr 03 2016 02:34
hmm 3.30 am. don’t think the time helps with me trying to understand things :-)
i better get some sleep. if you find some time to make an example(s) would be great. i learn from those things much faster then just reading :-)
Jordan Halterman
@kuujo
Apr 03 2016 02:38
I have some...
Anyways, we wouldn’t expect extensions to ZooKeeper to support messaging and replication, nor should we expect it of Atomix. ZooKeeper provides low level abstractions for building that crap, and Atomix provides a higher level abstractions for the same thing. Any serious messaging or replication framework should be a separate project, and Atomix should be left to mature.
I think it’s outside of the scope of the project. It just needs to be done separately. Atomix should only ever deal with atomic operations within the Raft consensus algorithm. Things that rely on the consistency of Atomix but add features on top of it should be in a separate project. And seriously scalable messaging and replication is that.
but I’m fine with doing it and working on it in a separate project
Jonathan Halterman
@jhalterman
Apr 03 2016 02:41
how would you feel about some of this stuff if not for the atomix-all dependency on it?
er, atomix
Jordan Halterman
@kuujo
Apr 03 2016 02:42
it should be in a separate project
it has nothing to do with the dependency
nothing is lost by putting it in a separate repository and calling it something else as a user of Atomix
but I think something is lost in Atomix by broadening the scope of the project
Jonathan Halterman
@jhalterman
Apr 03 2016 02:42
school bus
Jordan Halterman
@kuujo
Apr 03 2016 02:43
haha
Jonathan Halterman
@jhalterman
Apr 03 2016 02:43
good points
well argued sir
Jordan Halterman
@kuujo
Apr 03 2016 02:43
I just want Atomix to sit and mature that’s all
I wanted to do that stuff up until recently
but I’ve changed my opinion I suppose
Jonathan Halterman
@jhalterman
Apr 03 2016 02:43
mmm
Jordan Halterman
@kuujo
Apr 03 2016 02:44
bah I gotta hack this stuff out
to be continued
Richard Pijnenburg
@electrical
Apr 03 2016 10:07
@kuujo i meant some examples for my use case but fair enough :-) I’ll have to do more reading on this today.
Jordan Halterman
@kuujo
Apr 03 2016 10:15
I know I know
It won't have much beyond ``nextComponent.executor().execute(() -> nextComponent.run()) that's basically it. The rest is building the classes. Each component just has a reference to the next component and the next component's executor
Ugh messed that up
Typing on my phone
Richard Pijnenburg
@electrical
Apr 03 2016 10:17
hehe np
Jordan Halterman
@kuujo
Apr 03 2016 10:18
This can even be done with one class that implements Runnable. Inside the run method, do your think then execute nextComponent
Really the rest is just defining interfaces to make it pretty
Ugh
Richard Pijnenburg
@electrical
Apr 03 2016 10:19
hmm yeah. i want to make it as easy and clean as possible for the plugins.
preferably i don’t want the plugins to have any knowledge how it works on the background. they should just implement some methods that get called
Jordan Halterman
@kuujo
Apr 03 2016 10:27
Of course. That's easy. Just define interfaces. Interfaces define a method for plugins to perform operations. Implementations define what happens when those operations are performed.
Even if you use the basic pattern above with Runnable, plugins still don't have to know anything about where data is going or what is happening
By definition, if the plugins are not defining Runnable then they're abstracted from those details. It's just dependency injection. The manager provides the plugin with interfaces it uses for input and output, or the plugin can implement interfaces, or both
Richard Pijnenburg
@electrical
Apr 03 2016 10:31
for the plugins i just want them to have a run, init and shutdown method. what happens underneath they shouldn’t care
Jordan Halterman
@kuujo
Apr 03 2016 10:32
But run has to do something. Somehow its output has to go somewhere. Is that a return value?
what does the signature of run look like?
how does the plugin know what the input is, and how does it define what the output is?
Richard Pijnenburg
@electrical
Apr 03 2016 10:34
in the context of filters run should get a value in the method call, do somehting with it and submit / emit the result somewhere else.
Jordan Halterman
@kuujo
Apr 03 2016 10:36
public interface Plugin {
  void init(OutputCollector collector);
  void run(Object value);
  void shutdown();
}
public interface OutputCollector {
  void emit(Object value);
}
public interface Manager {
  void run();
}
Just have to define interfaces
These three interfaces define everything you need
plugins will not know anything about where data comes from or where it goes
public class TheOutputCollector implements OutputCollector {
  private final Executor executor;
  private final Plugin next;

  public TheOutputCollector(Executor executor, Plugin next) {
    this.executor = executor;
    this.next = next;
  }

  @Override
  public void emit(Object value) {
    executor.execute(() -> next.run(value));
  }
}
The OutputCollector is what knows where data goes
Richard Pijnenburg
@electrical
Apr 03 2016 10:39
executor is an instance of the Executors.newFixedThreadpool for example?
Jordan Halterman
@kuujo
Apr 03 2016 10:39
something that implements Executor… that’s one example
Richard Pijnenburg
@electrical
Apr 03 2016 10:39
okay
and next is an instance of the plugin it self
Jordan Halterman
@kuujo
Apr 03 2016 10:40
public class SomePlugin implements Plugin {
  private OuptutCollector output;

  @Override
  public void init(OutputCollector output) {
    this.output = output;
  }

  @Override
  public void run(Object value) {
    if (value == null) {
      output.emit(value);
    }
  }

  @Override
  public void shutdown() {
    // Do stuff
  }
}
Jordan Halterman
@kuujo
Apr 03 2016 10:45

To put it all together, the manager constructs a pipeline from back to front. The last component, then second to last, then third to last, back to the input.

List<Plugin> plugins = …;
Plugin next = null;
for (int i = plugins.size() - 1; i >= 0; i—) {
  OutputCollector output = new TheOutputCollector(Executors.newSingleThreadExecutor(), next);
  next = plugins.get(i);
}

Then the manager is constructed with the first plugin which is the only one it needs to run

Manager manager = new TheManager(next);
manager.run();

TheManager runs the first plugin, which runs the second, which runs the third, etc:

public class TheManager implements Manager {
  private final Plugin plugin;

  public TheManager(Plugin first, OutputCollector output) {
    this.first = first;
    first.init(output);
  }

  @Override
  public void run() {
    first.run();
 }
}

Something like that

The use of the OutputCollector allows a Plugin to emit more than one output in response to one input
I imagine there might be a separate Input interface though if inputs behave differently
Richard Pijnenburg
@electrical
Apr 03 2016 10:46
inputs do behave differently yeah
they have no input for example compared to filters
Jordan Halterman
@kuujo
Apr 03 2016 10:46
this is basically exactly the way Storm is designed
Richard Pijnenburg
@electrical
Apr 03 2016 10:46
outputs do have an input but don’t emit it anywhere else internally but to an external system
Jordan Halterman
@kuujo
Apr 03 2016 10:47
it has a spout and bolt. Both spouts an bolts are provided an OutputCollector to emit messages. etc
Richard Pijnenburg
@electrical
Apr 03 2016 10:48
possibly could abstract the manager a bit further with like add method to add plugins to the pipeline?
Jordan Halterman
@kuujo
Apr 03 2016 10:50
Yeah... If it should be mutable. But it doesn't seem like it should. What should probably exist is a mutable configuration for pipelines that's passed to the manager, or a ManagerBuilder that builds an immutable manager
Richard Pijnenburg
@electrical
Apr 03 2016 10:51
Yeah indeed. At some points the pipeline gets a new config, so it needs to shutdown, and restart with the new config
that for loop, where does it fit in with the manager code? bit confused since its separate
Jordan Halterman
@kuujo
Apr 03 2016 10:51
Which for loop?
Oh I see
Richard Pijnenburg
@electrical
Apr 03 2016 10:52
with the output collectors
Jordan Halterman
@kuujo
Apr 03 2016 10:52
Well... That could be in the constructor if you passed e.g. a List<Plugin> to TheManager, or it could be in TheManagerBuilder for instance.
Richard Pijnenburg
@electrical
Apr 03 2016 10:53
Ah okay.
Jordan Halterman
@kuujo
Apr 03 2016 10:53
I missed a step calling plugin.init(output) in that loop
Richard Pijnenburg
@electrical
Apr 03 2016 10:54
output plugins won’t have the output collector since they send data externally. so that would be a separate call i guess? since i can have multiple outputs.
Jordan Halterman
@kuujo
Apr 03 2016 10:54
It's pretty rudimentary but this is a general way to go about it. In working on Vertigo, the configuration part was far more time consuming than the actually message passing
Richard Pijnenburg
@electrical
Apr 03 2016 10:54
which makes it more confusing since a single event should go to all of them
Jordan Halterman
@kuujo
Apr 03 2016 10:55
Yeah see it's just the configuration and building the objects that takes some figuring out
If a single event can go to multiple outputs, then the OutputCollector implementation can have a list of outputs and loop through them to send one event to all outputs
It's just holding a meaningful configuration and then building all the OutputCollectors for each component that takes some tweaking
Richard Pijnenburg
@electrical
Apr 03 2016 10:56
Hmm yeah
I’ll try to build something with the examples you gave me. see if i can make something work.
Richard Pijnenburg
@electrical
Apr 03 2016 10:58
It all makes way more sense to you then to me at this point :p
Jordan Halterman
@kuujo
Apr 03 2016 10:59
It's a bit different than this. You add a list of components and create connections between them. When the network is deployed, that configuration is turned into a bunch of actual Connections
Indeed
Same principle though
4am and time for sleep
Richard Pijnenburg
@electrical
Apr 03 2016 11:00
oops. yeah indeed.
sleep well and thank you for the details
See if i can make something work with it :-)
Jordan Halterman
@kuujo
Apr 03 2016 11:01
:+1:
Roman Pearah
@neverfox
Apr 03 2016 16:31
This message was deleted
Things I hope to do with something like Atomix: a) a distributed leaky bucket for limiting workers making API calls, b) a distributed pool of IPs, with mutually-exclusive and rate-limited locks.
Roman Pearah
@neverfox
Apr 03 2016 16:38
I assume it's possible to set the types for a Resource to that of another Resource? For example, could you legitimately have a DistributedSet<DistributedLock>?
Richard Pijnenburg
@electrical
Apr 03 2016 16:39
The lock by definition is distrbuted, but not sure what you propose is possible
but @kuujo knows way more :-)
but i reckon he’s still sleeping.
Roman Pearah
@neverfox
Apr 03 2016 16:40
Right but I want to create a defined set of them that you can obtain all at once, as a set.
So you could iterate over it trying to obtain a lock
A "pool" of locks
Or maybe I want a map of locks, so I could look a lock up by key, for example.
Richard Pijnenburg
@electrical
Apr 03 2016 16:42
hmm good one.
Roman Pearah
@neverfox
Apr 03 2016 16:42
Clearly, nothing about the API seems to prevent it, but I don't know if that would break some of the guarantees
I'm also trying to figure out how to set something up with a good service discovery mechanism. I think I'll have to have some kind of controller cluster that manages the resources to a certain configured spec, rather than having clients create any of the resources. They would just use them.
Richard Pijnenburg
@electrical
Apr 03 2016 16:44
How do you mean?
Roman Pearah
@neverfox
Apr 03 2016 16:45
I mean that a lot of the examples show creating resources with getWhatever, but really I want all of the resources pre-created and then put into a discovery framework.
Clients won't be creating IPs in the pool. They'll already be pre-defined.
So I guess I'll need a "client" that's in charge of that.
Still trying to understand the difference between a client and a server in this model.
Richard Pijnenburg
@electrical
Apr 03 2016 16:47
well, the master can take care of that. if you provide a list of IP’s for example. the master process can define where they run.
i’m having the same problem. I’m creating sort of a processing pipeline and the master has to control which client gets what config
server in this case is an atomix replica. one that holds state. a client is something that can communicate with the cluster
the client can also register resources.
Roman Pearah
@neverfox
Apr 03 2016 16:48
That's what I thought
Richard Pijnenburg
@electrical
Apr 03 2016 16:49
like i have 3 clients and all 3 define the same lock. one of them is able to register it at that time. if that client dies, an other one will try to allocate the lock
Roman Pearah
@neverfox
Apr 03 2016 16:49
So in production terms, I would want to create a cluster of machines, say 3-5 EC2 instances or ECS containers that would run 24/7/
Richard Pijnenburg
@electrical
Apr 03 2016 16:49
yeah.
Roman Pearah
@neverfox
Apr 03 2016 16:50
I could then right more specific "clients" that have knowledge of what resources need to be created, and they would manage them.
Then I would have further "clients" that simply use those resources in some business logic to do work.
That way the cluster is generic.
Richard Pijnenburg
@electrical
Apr 03 2016 16:50
in my case i let the master decide what config a client gets and sends that to the client. that one then builds up all the resources.
Roman Pearah
@neverfox
Apr 03 2016 16:51
The way Zookeeper is generic.
What do you mean by config?
Are you saying that you're using Atomix to do distributed configuration management?
Richard Pijnenburg
@electrical
Apr 03 2016 16:51
what kind of processes it needs to run. including any atomix resources.
yeah, as part of it indeed
Roman Pearah
@neverfox
Apr 03 2016 16:52
I see
Richard Pijnenburg
@electrical
Apr 03 2016 16:52
that’s at least the idea :-)
But i’m a rubbish java programmer so not having much luck
Roman Pearah
@neverfox
Apr 03 2016 16:52
I'm basically trying to just provide distributed datatypes to an Onyx pipeline, so that my code can be written as if it weren't distributed.
Richard Pijnenburg
@electrical
Apr 03 2016 16:53
okay :-) i’m trying to rewrite an application called Logstash
into a fully distributed version
But i’m coming more from the Operations world and ruby.
java is a whole new thing for me
Roman Pearah
@neverfox
Apr 03 2016 16:54
Java's not really my thing either, being a functional guy, but in this case, I'm in a JVM language, so I have access to things like Atomix with interop.
Richard Pijnenburg
@electrical
Apr 03 2016 16:55
Im trying to build a processing pipeline now with some info @kuujo gave me but can’t make it to work :-( frustrating
Roman Pearah
@neverfox
Apr 03 2016 16:56
sorry
Maybe you should check out Onyx (since you're already diving into unfamiliar languages)
It's a processing pipeline framework
What it lacks though is any opinion on how you coordinate state across nodes.
Thus Atomix is something I'm looking into.
Richard Pijnenburg
@electrical
Apr 03 2016 16:57
I see :-)
Roman Pearah
@neverfox
Apr 03 2016 16:59
Clojure used to have a project called Avout, that attempted to make distributed versions of Clojure's atoms and refs, but it was abandoned because I think it had correctness concerns. Probably didn't fully think through things like Raft.
Richard Pijnenburg
@electrical
Apr 03 2016 17:00
clojure made even less sense to me then Java. that’s why i avoided that at all.
Richard Pijnenburg
@electrical
Apr 03 2016 17:14
@kuujo either i completely messed up your tips or i really can’t make it to run. I’m able to run the first plugin ( an input ) which should emit a string, but after that nothing happens :-(
Richard Pijnenburg
@electrical
Apr 03 2016 17:30
Ah. found out that the output queue its trying to send to is it self.
Richard Pijnenburg
@electrical
Apr 03 2016 17:56
@kuujo got it working i think by the looks of it. but required some hacking around.
I put a random sleep in my filter plugin to simulate a slow filter and seems to nicely continue to work on the other threads for events.
Jordan Halterman
@kuujo
Apr 03 2016 18:37
:clap:
Richard Pijnenburg
@electrical
Apr 03 2016 18:37
Trying to do some refactoring now to make it a bit more dynamic.. very painful :p
Richard Pijnenburg
@electrical
Apr 03 2016 18:49
@kuujo
        public void pluginList(List<Plugin> plugins) {
            Plugin next = null;
            Plugin first = plugins.get(0);

            System.out.println("First plugin is: " + first);
            for (int i = plugins.size() - 1; i >= 0; i--) {
                PluginOutputCollector output = new PluginOutputCollector(Executors.newFixedThreadPool(10), next);
                next = plugins.get(i);
                next.init(output);
                this.last = output;
            }
            ExecutorService executor = Executors.newFixedThreadPool(10);
            first.init(this.last);
            executor.execute(() -> first.run());
        }
the list was instances of the plugins
this is how i got it to work eventually
the input is ran separately from that loop
Roman Pearah
@neverfox
Apr 03 2016 18:50
@kuujo Is there a way to add members to an already running replica?
The only way it appears you can bootstrap a system is by knowing all of addresses you plan to launch ahead of time.
Which means you'd have to shut the replica down to scale it.
Richard Pijnenburg
@electrical
Apr 03 2016 19:01
@neverfox there should be a way. i know there was an issue about it just can’t find it now :-(
Richard Pijnenburg
@electrical
Apr 03 2016 19:09
@kuujo Now wondering how to implement conditionals in there as well :p lol like in the LS config in a single pipeline i can tell it to do a filter only if a field has value x for example..
Roman Pearah
@neverfox
Apr 03 2016 19:20
My first attempt the get a DistributedValue:
SerializationException cannot serialize unregistered type: class io.atomix.variables.state.ValueCommands$Get  io.atomix.catalyst.serializer.Serializer.writeByClass (Serializer.java:895)
Jordan Halterman
@kuujo
Apr 03 2016 19:20
This is a bug in the current release. It's fixed in master
Will be released hopefully todsy
Roman Pearah
@neverfox
Apr 03 2016 19:21
:+1:
Roman Pearah
@neverfox
Apr 03 2016 19:26
So @kuujo are you interested in having someone help get the Clojure library complete and up-to-date?
Jordan Halterman
@kuujo
Apr 03 2016 19:26
You can add a replica to a running cluster. We've been trying to figure out how to make the cluster configuration API more elegant. It's a bit unintuitive because of the constraints of Raft, and we're looking at another way to go about it. But I digress. If you start a three node cluster with each node listing it's local and the other two remote addresses in its configuration, you can add a fourth node to that cluster by simply pointing the fourth node at the first three. The fourth node technically only needs to point at one live member of the cluster. The decisions on whether to join the cluster or form a new cluster are transparent to the user. When the fourth node is started, it will send a join request to the remote nodes in its configuration. The new member will be logged and replicated through the leader to all the existing nodes.
Totally!
@jhalterman made that, but it was really just made for help with Jepsen tests
Hasn't really been seriously developed
Roman Pearah
@neverfox
Apr 03 2016 19:27
I'd be interested. Would be nice to go whole hog with it.
Jordan Halterman
@kuujo
Apr 03 2016 19:27
Definitely
Roman Pearah
@neverfox
Apr 03 2016 19:28
I'm glad it wasn't just me feeling the cluster config was awkward.
Jordan Halterman
@kuujo
Apr 03 2016 19:28
Yeah it is totally
Roman Pearah
@neverfox
Apr 03 2016 19:29
Is three the minimum before you can start doing what you suggest?
How did the Jepsen tests go? Are they published anywhere?
Jordan Halterman
@kuujo
Apr 03 2016 19:32
Yeah Jepsen tests have been passing for many months: http://github.com/atomix/atomix-jepsen Have never actually posted passing results anywhere... Haven't run them in a while so it's possible tests could be broken against master
Roman Pearah
@neverfox
Apr 03 2016 19:32
:clap:
Jordan Halterman
@kuujo
Apr 03 2016 19:37
Maybe you can give some input on what changes should be made. So, all that's really needed is for some cluster to be defined so that more nodes can be added to it. There are two ways to go about it. First, what Copycat/Atomix do is the first time the cluster is started, it uses that as the initial cluster configuration. So, you start a three node cluster and each node lists all three members (the initial configuration). When a fourth node is added, it sends a join request to the existing cluster. When a fifth node is added, it sends a join request, etc. This is safe so long as only one replica is added at a time after the initial bootstrap of the cluster since once at least one member is running, a single partitioned node cannot elect itself leader (any node added to the cluster always knows about at least one other node that's already a full member of the cluster and knows where the leader is). The alternative way to do it is with a single bootstrap node, and this is what I'm inclined to do in Atomix now. Essentially, when a cluster is started, a single replica is started with a bootstrapflag indicating that it can form a new cluster. Then, additional replicas are added to the cluster by pointing at the bootstrap node (or really any other member of the cluster). There's actually a third way, and that's to do it in the replica API. Expose separate start and join methods for bootstrapping a cluster and joining an existing cluster respectively. I actually think this may be the most obvious option and I'm rather fond of it.
Any of those changes can be made in quick order and will be better than what exists now :-(
Roman Pearah
@neverfox
Apr 03 2016 19:38
Yeah a single node starting point is what I was thinking, given how I've seen other systems.
Wasn't sure if it was possible.
I think Zookeeper and Consul, for example, do something like that
I love to be able to start a single node, have it register itself with a service discovery registry and then other nodes started later can find it and join.
Jordan Halterman
@kuujo
Apr 03 2016 19:47
Yeah that's an easy change to make
There's not much technical difference really. Currently, you just start a group of bootstrap nodes and point additional nodes at that cluster rather than starting a single bootstrap node and pointing additional nodes at that node. But it has proven to be confusing for people
Roman Pearah
@neverfox
Apr 03 2016 19:50
It's not so much confusing as that it's its own form of distributed coordination ;)
Currently the human is the leader lol
Jordan Halterman
@kuujo
Apr 03 2016 19:50
Right
ZooKeeper lists the full configuration on each node with a separate node ID file indicating the ID of the local node. Reading the Consul docs, it seems they're replacing the single bootstrap node approach with something more like a bootstrap cluster.
With --bootstrap-expect
As opposed to the manual bootstrap which starts a single bootstrap node: https://www.consul.io/docs/guides/manual-bootstrap.html
Roman Pearah
@neverfox
Apr 03 2016 19:55
That's cool.
Jordan Halterman
@kuujo
Apr 03 2016 19:56
The bootstrap behavior is still achieved in Atomix simply by starting a one node cluster, and that's all a bootstrap flag will do. When a one node cluster is started, the node can become leader since 1/1 is needed to elect a leader. Additional nodes pointed at it will join since they can't by themselves form their own cluster
Roman Pearah
@neverfox
Apr 03 2016 19:57
That seems logical. Is there a reason you think Consul went that route? Just to discourage single node clusters?
Jordan Halterman
@kuujo
Apr 03 2016 19:59
Yeah I'm not sure. I was wondering the same thing.
Roman Pearah
@neverfox
Apr 03 2016 20:00
Yeah, it looks like Zookeeper works similarly to Atomix now.
Jordan Halterman
@kuujo
Apr 03 2016 20:00
What about bootstrap() and join() methods rather than a single bootstrap node? That seems nice and I think more obviously allows either option. Maybe like bootstrap(Address... cluster) and join(Address... cluster)
Roman Pearah
@neverfox
Apr 03 2016 20:01
One thing to think about is how to get the sanity check / a-ok that the cluster is ready to accept clients.
I suppose that's one reason for the "expect 3" notion
So the cluster doesn't give a "ready" state until it reaches the desired size.
Jordan Halterman
@kuujo
Apr 03 2016 20:02
Ahh yeah that's interesting
Roman Pearah
@neverfox
Apr 03 2016 20:03
the method approach seems interesting
I'd have to see it to wrap my head around it
Jordan Halterman
@kuujo
Apr 03 2016 20:06
Effectively, you're saying bootstrap a cluster by forming a configuration with the provided Addresses. If no Address is provided then the local node is bootstrapped and elects itself leader so others can join. If some set of Addresses are provided, they form a cluster and elect a leader. join() requires some set of Addresses (at least one active member) through which it can join the cluster
Richard Pijnenburg
@electrical
Apr 03 2016 20:10
@kuujo what did you think of the code paste i did? trying to find a way to make it more dynamic
Jordan Halterman
@kuujo
Apr 03 2016 20:12
I pretty partial to bootstrap and join myself. I'm gonna throw together a PR
One sec
Roman Pearah
@neverfox
Apr 03 2016 20:15
I like it
Jordan Halterman
@kuujo
Apr 03 2016 20:15
I think it looks nice :-)
Richard Pijnenburg
@electrical
Apr 03 2016 21:28
looks nice? haha. its ugly. and to static
Jordan Halterman
@kuujo
Apr 03 2016 21:32
Hey... Build something simple and iterate iterate iterate
Richard Pijnenburg
@electrical
Apr 03 2016 21:32
hehe true :-)
I’m just to impatient :p lol
Richard Pijnenburg
@electrical
Apr 03 2016 22:00
Its hard to improve something if you miss the context and knowledge.
that for loop you created in your example works, but then i try to split it out into 3 separate loops for type of plugins and I’m unable to get it to work at all :(
Jordan Halterman
@kuujo
Apr 03 2016 22:02
what types of plugins are there?
input, filter, and output?
Richard Pijnenburg
@electrical
Apr 03 2016 22:02
yeah. there are also codecs but they are special, they modify the data stream inside an input or output. so they are not that much involved yet
Jordan Halterman
@kuujo
Apr 03 2016 22:02
it’s probably going to have to become some sort of recursive algorithm if events can be emitted to multiple outputs
Richard Pijnenburg
@electrical
Apr 03 2016 22:03
There can be multiple inputs, going through filters and then to multiple outputs
Jordan Halterman
@kuujo
Apr 03 2016 22:03
the first thing that’s needed is a configuration API to express all the use cases for a pipeline, and then build an algorithm to turn that into OutputCollectors and what not
it takes a NetworkConfig and turns it into a NetworkContext which defines connections between components, and that’s used to create inputs and outputs
A DefaultOutputCollector gets an OutputContext that tells it what to do with messages: https://github.com/kuujo/vertigo/blob/master/core/src/main/java/net/kuujo/vertigo/io/impl/DefaultOutputCollector.java
Jordan Halterman
@kuujo
Apr 03 2016 22:10
I think I almost got the bootstrap and join methods done in Copycat
Richard Pijnenburg
@electrical
Apr 03 2016 22:10
nice :-)
Jordan Halterman
@kuujo
Apr 03 2016 22:10
I like it much more… just gonna go through and make sure there are no safety issues with how it’s implemented
hmm bunch of issues to go through too ugh
Jordan Halterman
@kuujo
Apr 03 2016 22:28
K I think this is safe... Effectively, it just wraps the existing cluster. But I think it is a big improvement to what was there before. When a node is bootstrapped, the cluster stores the provided configuration which includes the local member. When tea node is joined, the cluster stores the provided configuration excluding the local member. The local server is then transitioned to its appropriate state according to the Member.Type and begins attempts to join the cluster. If the cluster is being bootstrapped, those attempts will fail and the election will timeout and the local node will start a new election. If it's a single node (bootstrap()) was called with no arguments), it will immediately elect itself leader. If it's a cluster, they'll elect a leader. If the node is joining but is partitioned from the leader, any election timeout and attempt to get elected leader will fail since no existing node will grant a vote. If join is called with only the local node in the configuration, the join will fail since there are no nodes to join.
Richard Pijnenburg
@electrical
Apr 03 2016 22:30
that sounds pretty safe yeah
I’m gonna grab some sleep. alarm goes off at 6
Catch you later bud.