These are chat archives for atomix/atomix

18th
Feb 2016
Mike Hearn
@mikehearn
Feb 18 2016 14:41
Hi, couple of questions
1) Any performance info anywhere? I couldn't find any
2) Does DistributedMultiMap preserve the order of (key, value) insertions? i.e. if I insert three values with the same key, do I always get back those three values in the same order I inserted them, and if so, is this documented/guaranteed anywhere? DMM doesn't seem to have any javadocs, really
Richard Pijnenburg
@electrical
Feb 18 2016 14:44
Hi @mikehearn there are no real perf tests yet. Did one with the task queue a while ago. I know that a perf test was done with the internal log part which could do about 100 actions / sec.
Now things have stabilised doing perf tests make sense indeed
100 should be 100k
Stupid auto correct.
Mike Hearn
@mikehearn
Feb 18 2016 14:45
haha, that sounds a bit better yeah
i'm interested in the perf of DMM when using a raft cluster in which nodes are a few milliseconds apart
Richard Pijnenburg
@electrical
Feb 18 2016 14:46
For multi map. Not sure if there is a test that tests that situation.
You could write a small test for it and try it out.
Jordan Halterman
@kuujo
Feb 18 2016 14:47
Hey! I'm here...
Mike Hearn
@mikehearn
Feb 18 2016 14:47
well, yeah, I could, but I'd be hoping that for a distributed map service the documentation would say :) i'm currently exploring options, I haven't actually used any code or downloaded atomix yet
Richard Pijnenburg
@electrical
Feb 18 2016 14:48
I'm afraid I won't be behind a computer until late tonight ( UK time ) to try it out.
Mike Hearn
@mikehearn
Feb 18 2016 14:48
there's no hurry
it might be a good idea to document the precise guarantees it provides in the javadoc, but beyond that, don't treat it as a priority because of me
Jordan Halterman
@kuujo
Feb 18 2016 14:51
Yeah I agree, there is more work that needs to go into the Javadoc.
So, on the order of DMM values, there are not currently guaranteed, but I can add that guarantee very easily as a configuration option :-) Currently, DMM values are stored in a HashMap in the state machine, but that could be changed to a LinkedHashMap to maintain insertion order based on the configuration of the map.
Mike Hearn
@mikehearn
Feb 18 2016 14:54
thanks
good to know
Jordan Halterman
@kuujo
Feb 18 2016 14:56
This is good to know for me too. Still in the process of documenting those things. DMM is one of the resources that hasn't really been documented. Would be good to make the order configurable in multimap and map.
Jordan Halterman
@kuujo
Feb 18 2016 15:07
On performance testing, I'm working on some tests on EC2 right now. TBH while the code is theoretically designed to be performant, all focus has been 100% on stability of the algorithms. The collections don't have the same overhead as queues. A write to a DMM only requires a single write, whereas message queues involve a write, read, and ack. Write performance to a map has been more like 10k/sec in my rudimentary tests, but I wouldn't call those legitimate performance tests as they weren't sustained for any long period of time. Read performance can be much, much faster than that depending on the consistency requirements because, like ZooKeeper, a read in Copycat/Atomix does not require hitting disk or replicating state and may not require any networking at all for sequential consistency.
map.get("foo", ReadConsistency.SEQUENTIAL) does a read from a single node normally without any network overhead (except to prevent state from going back in time when the client switches servers)
I have some time this morning. I'm going to document DMM and add some configuration options. Thanks! :-)
Mike Hearn
@mikehearn
Feb 18 2016 15:09
sweet!
by the way, something else that was not clear to me from the docs
is it reasonable for a DM or DMM to be larger than what fits in memory? is this a general sort of LevelDB style key->value disk map, or is a DM meant to be entirely in RAM
Jordan Halterman
@kuujo
Feb 18 2016 15:10
Entirely in memory right now. There are plans to support data structures larger than memory, but right now that's not the case. I'll document that limitation as well
Mike Hearn
@mikehearn
Feb 18 2016 15:10
ok
Jordan Halterman
@kuujo
Feb 18 2016 15:11
Thanks again!
Mike Hearn
@mikehearn
Feb 18 2016 15:11
finally, it wasn't clear to me what kind of commercial support Atomix has, if any. if it's just a hobby project, that's cool, but if it's supported by a company, that'd be useful to know too
Jordan Halterman
@kuujo
Feb 18 2016 15:16
No official commercial support. This was totally a hobby project, but these days I work on it with the help of employees of some commercially supported projects. Madam Jampani is a regular contributor/committer and has been a huge help as it has been used in ONOS for a while, and some other contributions come partly out of interest in using it at some companies (@jhalterman at HP), but that's the extent of corporate involvement. No corporate support is official, and I personally do this mostly on my free time :-)
Jordan Halterman
@kuujo
Feb 18 2016 15:25
By and large it's a passion project that happens to have gotten some interest lately. BTW there is very in depth documentation on the internals that's more complete than some of the other docs: http://atomix.io/copycat/docs/internals/
Mike Hearn
@mikehearn
Feb 18 2016 15:26
ok, thanks
Jordan Halterman
@kuujo
Feb 18 2016 15:27
Thanks for the pointers! Really helpful. It can be hard to imagine what's missing when you've been staring at something for too long. Always nice to have a fresh set of eyes that have the perspective of not knowing everything already.
@electrical I had to switch from late nights to early mornings now :-( haha
So not a morning person
But I'll probably be around here more in the mornings now
Jordan Halterman
@kuujo
Feb 18 2016 16:03
@mikehearn I was actually wrong. DistributedMultiMap already uses a LinkedHashMap for values. https://github.com/atomix/atomix/blob/master/collections/src/main/java/io/atomix/collections/state/MultiMapState.java#L78 Shame on me!
Mike Hearn
@mikehearn
Feb 18 2016 16:03
great!