These are chat archives for atomix/atomix

7th
Jan 2016
Richard Pijnenburg
@electrical
Jan 07 2016 00:00
and would save an extra dependeny then
*dependency
Jordan Halterman
@kuujo
Jan 07 2016 00:00
indeed
Jordan Halterman
@kuujo
Jan 07 2016 00:44
not too worried about performance, but running a quick performance test on the Copycat log since it’s been a while
Richard Pijnenburg
@electrical
Jan 07 2016 00:44
hehe okay ;-)
Richard Pijnenburg
@electrical
Jan 07 2016 00:52
I think for a first version of the clustering i should just let the whole pipeline exist on a single server.. and if i have multiple nodes, just copy that pipeline to multiple nodes.
assuming the pipeline is allowed to run there.
building the inter node communication from that point should be fairly easy
Richard Pijnenburg
@electrical
Jan 07 2016 00:58
uhg. 1 am. guess i better get some sleep. I'll talk to you later
Jordan Halterman
@kuujo
Jan 07 2016 01:00
alrighty
yeah this thing is fast enough I’d say
Jordan Halterman
@kuujo
Jan 07 2016 01:27
it’s writing entries at a rate that’s way too fast to even manage
wrote 1073741824 entries in 18898 milliseconds at 1775554.6618689809 per second
alright enough playing with that then
on to DistributedLog
the problem is, at that rate it creates segments faster than they can be compacted
haha
and reads in the Copycat log are generally done in O(1) time for uncompacted segments
Richard Pijnenburg
@electrical
Jan 07 2016 01:31
Is it possible to have separate logs for the cluster state and taks queue? Task queue doesn't need to be compacted since data should live there long anyway.
Jordan Halterman
@kuujo
Jan 07 2016 01:32
yeah
Richard Pijnenburg
@electrical
Jan 07 2016 01:32
Not sure if it will make any difference :)
Jordan Halterman
@kuujo
Jan 07 2016 01:33
Well, eventually disk will fill up. The Copycat log is designed to allow the user to specify when entries should be removed. Effectively, what I’m doing will make this much better. Logs will be separate from cluster state. That will allow concurrency anyways
Richard Pijnenburg
@electrical
Jan 07 2016 01:33
Ah nice
Jordan Halterman
@kuujo
Jan 07 2016 01:33
we’ll see what I came up with in the morning
your morning hopefully
Richard Pijnenburg
@electrical
Jan 07 2016 01:34
Hehe. Sounds good :)
Now I'm really gonna get some sleep. Lol
Jordan Halterman
@kuujo
Jan 07 2016 01:35
alright later
Richard Pijnenburg
@electrical
Jan 07 2016 01:35
Later.
Richard Pijnenburg
@electrical
Jan 07 2016 10:56
I tried out PR #105 and made it a little bit faster when producing and consuming at the same time. its between 2-3k / sec
when only consuming its hoovering around 14-15k with some peaks to 20-25k but also lows to 5-10k
I do have to admit its with disk storage and its running on a VM. so that part might not be the fastest thing
Jordan Halterman
@kuujo
Jan 07 2016 10:58
Yeah I had the same experience. I don't think that PR is done. Didn't make as big of an impact as it should have. Just stuck it there so I didn't forget about it :-)
Richard Pijnenburg
@electrical
Jan 07 2016 10:58
hehe okay :-)
I do wonder how fast the message bus is. did you do any tests with that?
Jordan Halterman
@kuujo
Jan 07 2016 11:00
Nope haven't done that one
Richard Pijnenburg
@electrical
Jan 07 2016 11:00
okay
Jordan Halterman
@kuujo
Jan 07 2016 11:00
I can do it
Richard Pijnenburg
@electrical
Jan 07 2016 11:00
only if you have the time. otherwise i can setup an example.
my main concern with that one, is that there is no acking that the other side received it.. bit like UDP :p
Richard Pijnenburg
@electrical
Jan 07 2016 11:24
and other downside is that i need a port for each message bus. which can be slightly annoying
but can be worked with
because for every node that has to read from the same queue i need to setup a message bus i guess
btw, #77 can be closed i thinj
Jordan Halterman
@kuujo
Jan 07 2016 11:29
Request-reply is an ack. Communication in the message bus is bi-directional. The reason there has to be a separate server is because a DistributedMessageBus can be created on a node that has no server to share. AtomixClient only has a Client, but you can still create a message bus there, and with a message bus every node is a client and server. This is not great to use as a queue. The Kafka-esque resource I'm making right now will only require a server for the brokers and makes more sense as a queue.
Richard Pijnenburg
@electrical
Jan 07 2016 11:29
ahhh okay :-)
btw, in gitter i see that line under my name. and also on yours. what does that mean?
hmm. guess because i forked the repo
Jordan Halterman
@kuujo
Jan 07 2016 11:38
Maybe?
Dunno
Really, the message bus is designed for something like service oriented architectures, which is where I stole the idea from (Vert.x)
Richard Pijnenburg
@electrical
Jan 07 2016 11:40
Ah i see. okay
Richard Pijnenburg
@electrical
Jan 07 2016 11:59
uhg.. i hate the elasticsearch team :-(
they can be so ops unfriendly :-(
Jordan Halterman
@kuujo
Jan 07 2016 12:28
Really? How so?
Richard Pijnenburg
@electrical
Jan 07 2016 12:29
the plugin script when you do any action on a plugin, it also checks all installed plugins for version numbers.
so i install ES 2.0.0 and install 2 plugins that works fine.
then i upgrade ES to 2.0.1
then i try to uninstall one of the plugins and install it again to get the new version.
on the install it also checks the other plugin which is still for the older ES version and then fails the install
Jordan Halterman
@kuujo
Jan 07 2016 12:30
Haha
Richard Pijnenburg
@electrical
Jan 07 2016 12:30
so it forces me to uninstall all plugins and then install the new ones.
Jordan Halterman
@kuujo
Jan 07 2016 12:30
Ouch
Richard Pijnenburg
@electrical
Jan 07 2016 12:31
which is like impossible in puppet
and between 1.x and 2.x they also changed so many CLI flags its just no fun anymore
Jordan Halterman
@kuujo
Jan 07 2016 12:33
Lol
Richard Pijnenburg
@electrical
Jan 07 2016 12:33
they are the worst team to work with from an OPS / automation perspective.
one of the many reasons i'm looking for an other job
Jordan Halterman
@kuujo
Jan 07 2016 12:36
Good teams are hard to come by
Richard Pijnenburg
@electrical
Jan 07 2016 12:36
yeah
And i'm the only one doing all this automation stuff as well.. and my manager doesn't care either