These are chat archives for atomix/atomix

18th
Jan 2016
Jordan Halterman
@kuujo
Jan 18 2016 01:24
Indeed.
Richard Pijnenburg
@electrical
Jan 18 2016 08:46
still busy i see :-)
Jordan Halterman
@kuujo
Jan 18 2016 08:47
indeed
almost got Copycat tests passing reliably
Richard Pijnenburg
@electrical
Jan 18 2016 08:48
nice :-)
Jordan Halterman
@kuujo
Jan 18 2016 08:48
trying to get rid of all that non-determinism
Richard Pijnenburg
@electrical
Jan 18 2016 08:48
yeah. those kind of things are horrible about teting
*testing
Jordan Halterman
@kuujo
Jan 18 2016 08:53
Yep, it’s pretty tedious working through intermitent test failures (race conditions) but it’s getting a lot more stable every day and that’s all it needs and is definitely worth the effort.
Richard Pijnenburg
@electrical
Jan 18 2016 08:54
i totally agree
that's what good software devs do :-)
To bad we have so many crappy ones at ES.
I even had to setup my own CI env to get things stable since our infra team sucks.
Jordan Halterman
@kuujo
Jan 18 2016 08:56
Lol wow that sucks. I love ours
Richard Pijnenburg
@electrical
Jan 18 2016 08:56
They are now building their own CI system
Jordan Halterman
@kuujo
Jan 18 2016 08:57
Interesting... Why?
Richard Pijnenburg
@electrical
Jan 18 2016 08:57
because they are fools.. and rather build somethign them selves then re-use existing things
And of course in a language that almost nobody uses.
non invented here behaviour.
Jordan Halterman
@kuujo
Jan 18 2016 08:59
Haha
Indeed
Richard Pijnenburg
@electrical
Jan 18 2016 09:00
and the manager just allows it
one of the many reasons i'm leaving
Jordan Halterman
@kuujo
Jan 18 2016 09:03
That is pretty crazy. It's insane to try to build your own everything. That's a recipe for failure and indeed that's what Kyle Kingsbury's criticism of ES has been. The best teams use the insane amount of tools available in the wild that people have built for years
Richard Pijnenburg
@electrical
Jan 18 2016 09:03
indeed :-) I reuse pretty much everything i can get my hands on
that's my own CI setup
jenkins with docker
fully puppetized
also jenkins config is automated
The only thing i do manually once in a while is updating the servers :-)
Jordan Halterman
@kuujo
Jan 18 2016 09:17
Look at Copycat and Atomix. I spent 3 years writing and rewriting it while studying the Raft algorithm, and it took me a long time to fully understand all of its different components and develop the additional algorithms for log compaction, client communication, and wait-free coordination. If a new team were starting from where I did - with no knowledge of consensus algorithms - it would still take a long time to develop a fully featured system simply because of the complexity of the concepts and the time it takes to learn and understand them. I've only had one contributor thus far come along and contribute significantly with a good understanding the algorithms (Madan Jampani who was one of the authors of Dynamo and has been a huge help with Copycat). You should almost never create a new system without having exhausted existing open source options. For instance, at my work we're thinking about writing our own job scheduler because the ones we've been using for the last two years are still a PITA to work with, but if we do decide to write our own in will have been after years of struggling with existing open source options. And there are some things that are just insane to try to create on your own. A home grown consensus algorithm is one of those things. There are only a few known correct consensus algorithms in existence (Paxos, Raft, View-stamped Replication, ZAB) and they all took years to develop, are peer reviewed, and require an insane amount of expertise to understand and implement. That is why many years after the creation of Raft I believe Copycat is one of the most if not the most feature complete implementation of the Raft consensus algorithm in existence, and it's still not totally stable.
Richard Pijnenburg
@electrical
Jan 18 2016 09:20
I totally have to agree with that. Its something i see every day. There are smarter people out there that build amazing things because they know what they are doing.
Jordan Halterman
@kuujo
Jan 18 2016 09:28
That's actually why I do this on the side. My work didn't need a consensus algorithm. But I was interested in consensus algorithms, so I work on open source projects so I can satisfy my interest in something that would otherwise be ridiculous to do at work. No way I could spend 3 years at work wasting time learning about something that is just now being finished up. We have other cool shit to do that's much more beneficial in the short term, and that's what companies should focus on, especially startups like mine.
Jordan Halterman
@kuujo
Jan 18 2016 09:33
That's what this project has taught me I think. The amount of expertise that you gain and lessons that you learn from working on a long term project as complex as this are completely invaluable and can be gained only through practice. At that point it's just insane not to use what other people have spent years building.
Richard Pijnenburg
@electrical
Jan 18 2016 09:33
Exactly :-)
if someone takes the time and effort in building something, why not use that instead of ductaping your own thing :-) because it will always be worse
Jordan Halterman
@kuujo
Jan 18 2016 09:34
...if you're not building something novel that adds a lot more value
always
Richard Pijnenburg
@electrical
Jan 18 2016 09:35
indeed.
Jordan Halterman
@kuujo
Jan 18 2016 10:07
Well, looks like the tests are about there. Time to go take a nap. Later!
Richard Pijnenburg
@electrical
Jan 18 2016 10:07
nice! catch you later.