These are chat archives for atomix/atomix

18th
Dec 2015
Richard Pijnenburg
@electrical
Dec 18 2015 12:16
@kuujo Nice! Looking forward to the details about it. And yeah, reliability can be important depending on the data that's being processed. Within Logstash they are now trying to build a persistent queue in there, which when running with atomix might not be needed. LS requires speed and reliability which makes it extremely hard for any system to do
I can also always grab some EC2 instances at work ( Elastic ) to do some testing if needed :-)
Richard Pijnenburg
@electrical
Dec 18 2015 16:55
before i confuse my self.. if i want a client only its AtomixClient and if i want a server ( leader electable ) i need AtomixReplica ?
Jordan Halterman
@kuujo
Dec 18 2015 17:16
Here's the difference between the three types in Atomix: Atomix is the interface that allows you to create and operate on resources (distributed objects). Both AtomixClient and AtomixReplica are Atomiximplementations. The difference between the client and replica is the client is purely remote and does not store state on disk, but the replica both stores state and operates on it. Because of the quorum restrictions, you should only really create 3 or 5 replicas and the rest clients. Later versions will abstract away that 3/5 limitation. AtomixServer is purely a server which does not provide an interface for creating or operating on distributed objects, but which clients and replicas can still communicate with. So, there are two patterns: use either servers or replicas for storing state, and replicas or clients for operating on it.
Richard Pijnenburg
@electrical
Dec 18 2015 17:18
@kuujo ahh interesting. thank you.. If i want a leader/server but with a rest api built in which is able to modify things, i assume i need a replica then so it can communicate internally to the cluster
Jordan Halterman
@kuujo
Dec 18 2015 17:18
It sounds to me like a persistent queue would help a lot. You could use Atomix to make a persistent queue distributed/fault tolerant - that's effectively what my Kafka clone will do - but that could not even be needed if it's already made fault tolerant by other means.
Yep you're correct
that actually seems to be a really good use for the replicas
Richard Pijnenburg
@electrical
Dec 18 2015 17:20
so i would have a leader role and a node role.. the node role handles all the tasks i need to distribute so that the leaders have all cpu cycles to make sure the cluster is stable.
leader's would contain all the data for the cluster like the main config which then is distributed to each node depending on their role.
just to make things more difficult :p
a node can have 3 roles, inputs, outputs and filters
for the inputs i want to be able to have a lock in place for some things because i want only a single instance of a plugin running inside the cluster
Jordan Halterman
@kuujo
Dec 18 2015 17:23
That sounds amazing
Richard Pijnenburg
@electrical
Dec 18 2015 17:23
also looking to implement sort of a pipeline. for example a redis input, some filters and an output would be a single pipeline that functions completely independent from other pipelines
wish i knew more about java then i do :-)
I'm more a sysadmin / Ruby programmer\
Jordan Halterman
@kuujo
Dec 18 2015 17:26
Yeah it's a big change... But Java tends to avoid a lot of magic so it can be easier to figure out
Gotcha
Richard Pijnenburg
@electrical
Dec 18 2015 17:26
and mainly work on config management tooling :p ( puppet )
Jordan Halterman
@kuujo
Dec 18 2015 17:28
Learning is the best :-)
Richard Pijnenburg
@electrical
Dec 18 2015 17:28
hehe very true
Jordan Halterman
@kuujo
Dec 18 2015 17:30
I think I'll try to steal some EC2 instances this evening
Richard Pijnenburg
@electrical
Dec 18 2015 17:58
hehe okay. going to do those perf tests?
Hm, trying to find something like the ruby version of Thread so i can start a management thread for atomix so it doesn't mess up with other threads