These are chat archives for atomix/atomix
sync()? I mentioned there was a bug that was stopping it from going fast. I fixed that
asyncqueue uses that's an even worse case than what
new Storage("logs", StorageLevel.MAPPED)
DistributedWorkQueueconsumers seems to be an issue of implementation more than anything else. The reason the example producer is so much faster than the example consumer when both are running is because the producer writes 100 concurrent tasks to the cluster, and the consumer only consumes one at a time. The consumer acks one message and gets the next. So, when a producer and consumer are both running, the producer is simply using the majority of the bandwidth producing tens of thousands of tasks/sec. Thus, when you add a second consumer the first consumer does down more simply because there are more writes being spread across the cluster, and it's not linearly scalable. This implies that batching needs to be done on the consumer side to keep up with tasks being produced to the queue concurrently, or tasks need to be pipelined to consumers. I'm implementing the former today, but the latter is also totally feasible and may be more reliable.
Transport. It would just require a special
Transportthat wraps the
NettyTransportto allow it to be shared. But Atomix won't have its own sharding until version 2, and that will probably be in 2017. There are too many complexities to sharding to make it worth battling in Atomix at the moment. If someone hires me to do it I'll do it, but aside from that happening it will be a long time before its implemented. The problem is sharding implies a loss of consistency across shards. You have to implement something like a two-phase commit protocol to fairly safely handle creating and deleting resources that span multiple shards, and resources themselves have to understand sharding. Your best move if you need that sort of throughput is building an abstraction to use different types of messaging layers or simply use the
DistributedMessageBus, create your own shards, and use request-reply to ensure reliability. This is what I'm doing in the Kafka-esque example.
DistributedMessageBusdoesn't guarantee one message will get from point A to point B doesn't mean it can't be used to ensure it. Create a
DistributedMembershipGroupfor each partition and send messages to the leader of the group. The leader replicated messages to followers over the same message bus to make them fault tolerant, and the leader pushes messages to a consumer.