These are chat archives for atomix/atomix

6th
Jan 2016
Jordan Halterman
@kuujo
Jan 06 2016 04:01 UTC
been chasing the bug causing session events to slow down for a few hours
finally found it
Richard Pijnenburg
@electrical
Jan 06 2016 07:50 UTC
Oh nice!
Richard Pijnenburg
@electrical
Jan 06 2016 09:34 UTC
Been trying out the producer and consumer and using 'pv' to measure the rate.. its.. euhm.. disapointing :(
between 200 and 400 messages / sec
but when i stop the producer it goes up to something between 10k and 20k messages / sec
with peaks between 20 and 25k
and the consumer can run for quite a while after i stopped the producer, so that one is producing much faster then the consumer can run when its producing and consuming at the same time.
after a while ( after stopping the producer ) it stabilizes below 10k
Richard Pijnenburg
@electrical
Jan 06 2016 09:40 UTC
if i have 2 producers and 1 consumer for the same queue it sometimes gets even worse.. below 300 msg/sec
most of the time even below 200
Jordan Halterman
@kuujo
Jan 06 2016 10:20 UTC
Ugh come on gitter
Richard Pijnenburg
@electrical
Jan 06 2016 10:20 UTC
hehe
still awake? its like 2 am for you i think
Jordan Halterman
@kuujo
Jan 06 2016 10:21 UTC
@electrical Are you using sync()? I mentioned there was a bug that was stopping it from going fast. I fixed that
Richard Pijnenburg
@electrical
Jan 06 2016 10:21 UTC
i used the latest examples
updated everything and built it from scratch
Richard Pijnenburg
@electrical
Jan 06 2016 10:26 UTC
as far as i can see it uses the async thing
Jordan Halterman
@kuujo
Jan 06 2016 10:26 UTC
Yeah
The consumer slows down when there's a producer?
Richard Pijnenburg
@electrical
Jan 06 2016 10:27 UTC
yeah
Jordan Halterman
@kuujo
Jan 06 2016 10:30 UTC
Ahh I know the problem
I lied
It should most likely be producing and consuming at the rate you see the consumer moving when there no producer
Richard Pijnenburg
@electrical
Jan 06 2016 10:33 UTC
well, as soon as i stop the producer, the consumer keeps on going but at a much faster pace.
Jordan Halterman
@kuujo
Jan 06 2016 10:33 UTC
Something's causing it to be synchronous, and with the consistency level that the async queue uses that's an even worse case than what sync does.
I'll have to look at it in the morning after another long meeting, but considering the work that a consumer does by itself to pull messages off the queue and ack messages, the producer and consumer should both be able to move at that rate. Something in the logic when they're both active is probably preventing what you see the consumer doing when there's no producer. Sounds like maybe the consumer is waiting for more messages when it should be continuously getting them, which is what you see when the producer is killed.
Jordan Halterman
@kuujo
Jan 06 2016 10:38 UTC
I like that 10-20k number better
Richard Pijnenburg
@electrical
Jan 06 2016 10:39 UTC
10-20k is fairly nice yeah. wonder if it could be faster
multiple consumers didn't make it really faster
Jordan Halterman
@kuujo
Jan 06 2016 10:40 UTC
It can be made faster by batching for sure
But considering the consumer is committing 10-20k writes/sec (an ack requires a write to disk) that's pretty much exactly where you want Raft to be at least. It's a good place to start
Richard Pijnenburg
@electrical
Jan 06 2016 10:41 UTC
yeah defo.
it does slow down after a while
but can do some better speed tests i think
Jordan Halterman
@kuujo
Jan 06 2016 10:43 UTC
Performance is something that has been completely disregarded until now, but this is about the time to start improving it a bit, at least where there are big gains to be made.
Richard Pijnenburg
@electrical
Jan 06 2016 10:43 UTC
yeah defo
most important thing so far was to get it stable :-)
which it is
Jordan Halterman
@kuujo
Jan 06 2016 10:46 UTC
Copycat's internal log handles writing something like 80k average sized entries/sec to disk on my MacBook. BTW you can also use memory mapped files to get a huge performance increase or just memory if you're really brave
Richard Pijnenburg
@electrical
Jan 06 2016 10:46 UTC
ah good one. using the disk storage by default right?
Jordan Halterman
@kuujo
Jan 06 2016 10:46 UTC
Storage storage = new Storage(StorageLevel.MAPPED)
Yeah
Richard Pijnenburg
@electrical
Jan 06 2016 10:47 UTC
That has to be set in the leaders right?
Jordan Halterman
@kuujo
Jan 06 2016 10:47 UTC
Yeah when constructing an AtomixReplica or AtomixServer
Richard Pijnenburg
@electrical
Jan 06 2016 10:47 UTC
okay
in the leader example there is the args[0]
where do i add it to?
Jordan Halterman
@kuujo
Jan 06 2016 10:48 UTC
Ahh yeah that's the path
You can do new Storage("logs", StorageLevel.MAPPED)
Richard Pijnenburg
@electrical
Jan 06 2016 10:48 UTC
ah okay. second argument :-)
ah. and needed an other import it seems.
import io.atomix.copycat.server.storage.StorageLevel;
Jordan Halterman
@kuujo
Jan 06 2016 10:51 UTC
Alright gotta go take a nap
Richard Pijnenburg
@electrical
Jan 06 2016 10:51 UTC
okay. catch you tomorrow
Jordan Halterman
@kuujo
Jan 06 2016 10:52 UTC
I haven't tested them in a while, but IIRC memory mapped files were a big performance boost. The work queue might still be slow until I fix it though. But the consumer might speed up more just from that.
Adios
Richard Pijnenburg
@electrical
Jan 06 2016 10:52 UTC
laterz
Jordan Halterman
@kuujo
Jan 06 2016 21:13 UTC
So, the issue of DistributedWorkQueue consumers seems to be an issue of implementation more than anything else. The reason the example producer is so much faster than the example consumer when both are running is because the producer writes 100 concurrent tasks to the cluster, and the consumer only consumes one at a time. The consumer acks one message and gets the next. So, when a producer and consumer are both running, the producer is simply using the majority of the bandwidth producing tens of thousands of tasks/sec. Thus, when you add a second consumer the first consumer does down more simply because there are more writes being spread across the cluster, and it's not linearly scalable. This implies that batching needs to be done on the consumer side to keep up with tasks being produced to the queue concurrently, or tasks need to be pipelined to consumers. I'm implementing the former today, but the latter is also totally feasible and may be more reliable.
Jordan Halterman
@kuujo
Jan 06 2016 22:27 UTC
Actually implementing the latter. Batching is done internally by Copycat anyways, so no need to also batch outside of it. Pipelining implies batching
Richard Pijnenburg
@electrical
Jan 06 2016 22:51 UTC
Hi sorry, was playing a game :-)
ahh okay. any idea what the max bandwith totally is ? seems to be around 20 - 25k or something at least in my test
Richard Pijnenburg
@electrical
Jan 06 2016 22:57 UTC
it did peak to about 30k at some point
I'm very worried for setups when they need like 100k - 200k or even more
Richard Pijnenburg
@electrical
Jan 06 2016 23:05 UTC
Btw, switching to mapped storage didn't improve at all
Jordan Halterman
@kuujo
Jan 06 2016 23:08 UTC
Yeah totally. There's no way Atomix can do that. At that point you need a system that scale linearly, which consensus algorithms do not (unless you use some sort of sharding). I'm guess with a lot of optimization and memory mapped files it could get up to 40 or 50 or 60k/sec, but keep in mind that's the full capacity of the cluster. If you're using a bunch of resources then the total throughput of all writes from all resources will be limited to that number. Sharding is the only way to make it scale beyond that. You could conceivably do sharding by creating multiple Atomix clients/replicas that share the same Transport. It would just require a special Transport that wraps the NettyTransport to allow it to be shared. But Atomix won't have its own sharding until version 2, and that will probably be in 2017. There are too many complexities to sharding to make it worth battling in Atomix at the moment. If someone hires me to do it I'll do it, but aside from that happening it will be a long time before its implemented. The problem is sharding implies a loss of consistency across shards. You have to implement something like a two-phase commit protocol to fairly safely handle creating and deleting resources that span multiple shards, and resources themselves have to understand sharding. Your best move if you need that sort of throughput is building an abstraction to use different types of messaging layers or simply use the DistributedMessageBus, create your own shards, and use request-reply to ensure reliability. This is what I'm doing in the Kafka-esque example.
Richard Pijnenburg
@electrical
Jan 06 2016 23:10 UTC
the message bus would be much faster?
Jordan Halterman
@kuujo
Jan 06 2016 23:10 UTC
Just because DistributedMessageBus doesn't guarantee one message will get from point A to point B doesn't mean it can't be used to ensure it. Create a DistributedMembershipGroup for each partition and send messages to the leader of the group. The leader replicated messages to followers over the same message bus to make them fault tolerant, and the leader pushes messages to a consumer.
Yeah. It's just a TCP connection
Richard Pijnenburg
@electrical
Jan 06 2016 23:10 UTC
ahh okay
Jordan Halterman
@kuujo
Jan 06 2016 23:11 UTC
Direct TCP
Maybe I should actually write that example tonight
I'm curious myself
Richard Pijnenburg
@electrical
Jan 06 2016 23:11 UTC
hehe :p
if i would use that then i need some persistent queue somewhere that's fast enough
and still need some ack method i guess
hmm. it could be pretty fast if you use a pool of connections instead a single connection
without using the distributed work queue i'll need buffers on both sides for sending and receiving and build some kind of acking my self i guess
Richard Pijnenburg
@electrical
Jan 06 2016 23:16 UTC
and maybe have something like a thread pool for the consumers and producers if that's even possible :-)
would be nice if we could build something that doesn't require an external system to save the messages..
Richard Pijnenburg
@electrical
Jan 06 2016 23:23 UTC
would hate to use redis or something else for message queuing :-)
Richard Pijnenburg
@electrical
Jan 06 2016 23:30 UTC
form what i read in theory you could add acking in the messagebus perhaps? if its async it will send it, wait for the otherside ( consumer ) to do its thing, ack it and send the ack back to the sender ( producer )
doing that with multiple threads ( configurable threadpool ) might work
since it would all be outside of raft it should be much faster then i guess
Jordan Halterman
@kuujo
Jan 06 2016 23:33 UTC
yep
Richard Pijnenburg
@electrical
Jan 06 2016 23:33 UTC
i could have a local persistent queue on each node invidiually, which then is defo faster then having a central queue
think that's pretty much mandatory to increase the speed
Jordan Halterman
@kuujo
Jan 06 2016 23:36 UTC
that effectively what sharding would be
Richard Pijnenburg
@electrical
Jan 06 2016 23:36 UTC
yeah indeed
Jordan Halterman
@kuujo
Jan 06 2016 23:36 UTC
I may actually just make a fast messaging resource
tonight
maybe I’ll give it a shot
Richard Pijnenburg
@electrical
Jan 06 2016 23:37 UTC
okay :-)
Jordan Halterman
@kuujo
Jan 06 2016 23:37 UTC
or I will give it a shot
should be easy and interesting
Richard Pijnenburg
@electrical
Jan 06 2016 23:41 UTC
hmm. only issue then is that i'll have to get the remote node pull data instead of send it.. otherwise i cant have multiple nodes read from the same queue
to spread the load
unless i start multiple message busses to send to the nodes that read from the queue
that could work as well
not sure if this is interesting to use: https://github.com/bulldog2011/bigqueue
Richard Pijnenburg
@electrical
Jan 06 2016 23:49 UTC
according to the stats it can do about 150 - 160k messages / sec
but the guy hasn't maintained it in ages
@kuujo not sure if this is interesting: https://github.com/bulldog2011/luxun
Jordan Halterman
@kuujo
Jan 06 2016 23:53 UTC
another Kafka clone
Richard Pijnenburg
@electrical
Jan 06 2016 23:54 UTC
ah, i see it yeah
Jordan Halterman
@kuujo
Jan 06 2016 23:54 UTC
hmm
it has a package named “bigqueue"
are the two the same thing?
Richard Pijnenburg
@electrical
Jan 06 2016 23:54 UTC
bigqueue is the queuing library it uses for persisting the data i think
Jordan Halterman
@kuujo
Jan 06 2016 23:54 UTC
ahh gotcha
Jordan Halterman
@kuujo
Jan 06 2016 23:56 UTC
I think maybe it’s distributed but not fault tolerant
I was trying to figure out how it was distributed and don’t see any libraries like ZK client or anything
Richard Pijnenburg
@electrical
Jan 06 2016 23:56 UTC
no indeed
Jordan Halterman
@kuujo
Jan 06 2016 23:58 UTC
I’m gonna hack this together :-)
Richard Pijnenburg
@electrical
Jan 06 2016 23:58 UTC
hehe okay :-)
Jordan Halterman
@kuujo
Jan 06 2016 23:58 UTC
can even use Copycat’s log for memory mapped files
Richard Pijnenburg
@electrical
Jan 06 2016 23:58 UTC
assuming its fast enough :-)