These are chat archives for atomix/atomix

4th
May 2018
buko
@buko
May 04 2018 04:18
Does anybody have an example of how LeaderElection works in the new Atomix 2.0?
Jordan Halterman
@kuujo
May 04 2018 04:18
What do you mean by how it works? An example usage of the API?
buko
@buko
May 04 2018 04:20
yes. The old atomix had an examples folder with lots of examples. I'm looking at atomix2 now and I found this test https://github.com/atomix/atomix/blob/master/core/src/test/java/io/atomix/core/election/impl/LeaderElectionTest.java
it is not so easy to understand
buko
@buko
May 04 2018 04:25
hmm
Jordan Halterman
@kuujo
May 04 2018 04:31
Atomix atomix = new Atomix(“atomix.yaml”);
atomix.start().join();

LeaderElection<MemberId> election = atomix.getLeaderElection(“test”);

// Add an election listener
election.addListener(event -> {
  Leadership<MemberId> leadership = event.newLeadership();
  ...
});

// Attempt to become the leader
election.run(atomix.membershipService().getLocalMember().id());

// Get the current leadership
Leadership<MemberId> leadership = election.getLeadership();
BTW this is on the front page of the website
buko
@buko
May 04 2018 04:33
huh, thanks Jordan. totally missed that. When I go to http://atomix.io/docs/latest/user-manual/primitives/LeaderElection/ all I see is a blank page.
Jordan Halterman
@kuujo
May 04 2018 04:33
still working on the documentation
probably will do the primitives next actually
buko
@buko
May 04 2018 04:36
I'm evaluating atomix2. I see now that it has been released... 2.0 artifacts are in mvn central and it's good for production use?
Jordan Halterman
@kuujo
May 04 2018 04:38
Atomix 2.0 is mostly just the Raft implementation. It’s much, much more stable than Atomix 1 ever was. Atomix 2.1 is the finished product. The odd versioning was the result of needing to have a stable release of the new Raft implementation to depend on
I guess we could just as easily call this Atomix 3
buko
@buko
May 04 2018 04:39
2.0 doesnt include the primitives?
Jordan Halterman
@kuujo
May 04 2018 04:39
no
buko
@buko
May 04 2018 04:39
ok. but the primitives are mostly done in the 2.1 betas? are the apis stable?
buko
@buko
May 04 2018 04:53
btw, is anybody using atomix with jms? Our specific case is just a cluster of nodes processing messages sent to a jms queue. The only tricky part is we don't want to ack the jms message until its effects have been replicated.
I don't think we need full 2pc/XA for this, we might just treat this as a nested transaction. I saw atomix has something called a WorkQueue. Is this guaranteed? Might this be used for distributing replication tasks?
Jordan Halterman
@kuujo
May 04 2018 05:04

The primitives are done and have been in use in some cases for years. Most of the new code is just from API changes, various services built for managing the cluster and partitions, and the new primary-backup protocol. Hopefully next week will be a release candidate, and after that all APIs not annotated @Beta will be considered stable.

The work queue guarantees a task will be received and acknowledged by some process before it’s removed from the primitive. Whether that is sufficient depends on your architecture and what is meant by replicating its effects.

We actually use the primitives very heavily. They are stable. It’s the protocols that underly them that have been modified.
buko
@buko
May 04 2018 05:08
ok very cool stuff. Do you have an example of the WorkQueue api. The example on the front page is incomplete. The simplest replication strategy for us might be fort the leader to replicate the inbound message (an xml doc serialized as byte[]) to a majority of the cluster and for them to (eventually) write that to disk
once the inbound message has been replicated and acked by the other cluster members we can be reasonably sure if the leader falls down somebody else can take over.
Jordan Halterman
@kuujo
May 04 2018 05:42

This can get a little weird because adding an item to a work queue also replicates it to a majority of the cluster and writes it to disk already (if you’re using Raft), deleting it from the cluster once it’s acknowledged by a task processor. Unless you need items to be processed asynchronously and to persist through failures, you can just use regular messages to do the same thing. But I guess an argument could be made for using the work queue with the primary-backup protocol. That would essentially replicate it in memory until it’s processed by some other node. But the internal protocol is what replicates an item. The work queue then ensures that the item is eventually processed by one node.

If you just need something to be replicated for redundancy, the typically way to do that is using a data primitive like a map, set, value, etc. Those primitives can be replicated to as many nodes as you want using either Raft or primary-backup partitions

buko
@buko
May 04 2018 05:50
yeah, reading more into the docs it may seem like a map makes more sense. It solves the synchronization problem off the bat... when a new node joins the cluster it just gets the whole datastruct right? But are the data structures backed by any kind of persistence? We might want the ability to synchronize a new cluster member out-of-band (copy files from one host to another) and if a member needs to be restarted it shouldn't need to re-sync everything if it has some data stored on its local disk.
more importantly... we definitely need the data to be archived everyday at the end of business. I guess the big question is can we plugin to the distributed map and in another thread write everything to disk
ah I see the work queue doesn't distribute work to a cluster, you're saying it distributes work to one node in a cluster who must process and ack it?
Johno Crawford
@johnou
May 04 2018 05:56
That's the definition of distribute
Jordan Halterman
@kuujo
May 04 2018 05:57
Yeah
Johno Crawford
@johnou
May 04 2018 05:57
Haven't checked it personally but generally queue infers it will dispatch tasks to various nodes in the cluster
Jordan Halterman
@kuujo
May 04 2018 05:58
Indeed. A process acks an item and gets the next item. So generally a set of nodes or processes pull items off the queue, process, and acknowledge them
Johno Crawford
@johnou
May 04 2018 05:58
Topic on the other hand would be all interested nodes
Jordan Halterman
@kuujo
May 04 2018 05:58
Yep
@buko primitives can either be replicated using Raft - which uses a commit log on disk - or primary-backup - which just replicates changes in memory. Raft can tolerate restarts, primary-backup can’t. All nodes can share the same primitives, and the physical state can be stored on any node you want it to be stored on.
FYI I’m getting closer to figuring out wtf is hanging around in these tests
Johno Crawford
@johnou
May 04 2018 06:02
Thank goodness because it was driving me mad :smile:
buko
@buko
May 04 2018 06:05
ok thanks Jordan. will dig into it a bit more and see if I can do a poc.
Johno Crawford
@johnou
May 04 2018 07:21
@kuujo any luck? I see a bunch of changes already
buko
@buko
May 04 2018 07:22
@johnou just to be clear, I don't have to be a raft member -- I can be a client member -- to use the primitive types like map and set right?
Jordan Halterman
@kuujo
May 04 2018 07:22
nope
nope to @johnou
Johno Crawford
@johnou
May 04 2018 07:22
@buko yes
Jordan Halterman
@kuujo
May 04 2018 07:22
haha
Jordan Halterman
@kuujo
May 04 2018 09:08
Slowly making progress. Fixed a lot of little issues. Haven’t tracked down the biggest issue yet though
Johno Crawford
@johnou
May 04 2018 09:19
@kuujo well if stop is hanging
two potential problems
either a thread is actually hanging
thread dump would help, or the cf is not going to complete because a chain is broken somewhere
maybe exception is breaking it
i generally prefer whenComplete(Async) and handle(Async)
i did notice a service can get into a loop
not sure if you fixed that?
io.atomix.protocols.backup.proxy.PrimaryBackupProxy#execute(io.atomix.primitive.operation.PrimitiveOperation, io.atomix.utils.concurrent.ComposableFuture<byte[]>)
if (cause instanceof PrimitiveException.Unavailable) {
            threadContext.schedule(Duration.ofMillis(RETRY_DELAY), () -> execute(operation, future));
this line
Jordan Halterman
@kuujo
May 04 2018 09:27
Like I said, that was from the primary election bugs, but I already added limited retries to that method anyways
Johno Crawford
@johnou
May 04 2018 09:27
ah I missed that
Jordan Halterman
@kuujo
May 04 2018 09:28
It’s in the PR
The primary-backup client seems to be working well. Just need to spend another day tracking down whatever else is wrong
Johno Crawford
@johnou
May 04 2018 09:30
Retry operations on timeout in PrimaryBackupProxy. 40 minutes ago
i was putting together a wheelbarrow
:D
Jordan Halterman
@kuujo
May 04 2018 09:30
Also need to clean it up. Clients should be able to read from backups and read their own writes.
Haha
The Raft client and servers have had such a massive amount of work put into them. It will take a while for the primary-backup protocol to catch up even though it’s much simpler.
Johno Crawford
@johnou
May 04 2018 09:34
oh if you ever need to debug a node app
longjohn
saved my day
Jordan Halterman
@kuujo
May 04 2018 17:06
that’s the worst part about asynchronous code… the stack traces are almost useless
Jordan Halterman
@kuujo
May 04 2018 20:11
🤞
Jordan Halterman
@kuujo
May 04 2018 20:36
at the point now where one test seems to be consistently failing on Travis but not in my IDE