These are chat archives for atomix/atomix

28th
Feb 2017
Alex Atomei
@AlexAtomei1_twitter
Feb 28 2017 16:19

Hey guys, I have a silly question. Just started experimenting with atomix and I'd like to know how can I create a DistributedValue in the cluster? I have an instance of AtomixClient but when trying to create a new DistributedValue<String> it requires a CopyCatClient instance and when trying to use atomixClient.getValue(key).join().set(value); I get a ClassNotFoundException: io.atomix.variables.util.DistributedValueFactory

In the documentation for example there is this:
value.set("Hello world!", Duration.ofSeconds(1)).thenRun(() -> {
System.out.println("Value set with TTL of 1 second");
});

but how is that "value" object created? Thank you very much.

vishwas
@vishwass
Feb 28 2017 17:57
@kuujo I wanted to modify copycat project. I wanted to change the mechanism
Could you please guide me as to where in the code does commiting to log occurs ?
Jordan Halterman
@kuujo
Feb 28 2017 18:33

@vishwass how do you want to modify it?
Copycat is a complete implementation of the Raft consensus protocol: https://ramcloud.stanford.edu/~ongaro/thesis.pdf
Committing to the log really happens at different points on different servers depending on their state. The server states are here: https://github.com/atomix/copycat/tree/master/server/src/main/java/io/atomix/copycat/server/state
In general, the leader receives a command from a client, writes it to its log, replicates it, and commits it once the entry has been replicated to a majority of the cluster. Followers later commit the entry as well. Writing changes to the log and committing them are two different concepts. Actual commits occur in LeaderAppender by counting replicas. Committing an entry essentially just means setting commitIndex. Then they’re committed on followers when they receive AppendRequest.

If you want to understand how it works, you should start with the original Raft paper. Once you understand that, the Copycat server states become easy(er) to understand.

Jordan Halterman
@kuujo
Feb 28 2017 18:48

@AlexAtomei1_twitter how is the cluster set up? If you use the standalone server, you have to ensure the atomix-variables jar is added to the classpath and to the server’s configuration. TBH the standalone server is about to get a huge makeover because it’s terrible to work with.

So, typically what people do is add the atomix-all Maven dependency and create an AtomixReplica (or 3, or 5):

AtomixReplica replica = AtomixReplica.builder(new Address(“localhost”, 5000))
  .withTransport(new NettyTransport())
  .build();
replica.bootstrap().join();

You can bootstrap a full cluster by passing all the nodes to the bootstrap() method, or just add more replicas to the bootstrapped node:

AtomixReplica replica = AtomixReplica.builder(new Address(“localhost”, 5001))
  .withTransport(new NettyTransport())
  .build();
replica.join(new Address(“localhost”, 5000)).join();

Then you can either create a DistributedValue using the replica:

DistributedValue<String> value = replica.getValue(“foo”).join();

Or you can create a client and connect to your replicas:

AtomixClient client = AtomixClient.builder()
  .withTransport(new NettyTransport())
  .build();
client.connect(new Address(“localhost”, 5000), new Address(“localhost”, 5001)).join();

DistributedValue<String> value = client.getValue(“foo”).join();

When you call getValue on a replica or client, a command will be submitted to the cluster to create a new DistributedValue state machine on all the nodes. Then you’ll get back a DistributedValue that submits state machine commands via the wrapped CopycatClient.

It is possible to just create a DistributedValue using the constructor, but you have to have a Copycat cluster with the DistributedValue state machine to connect to.

One caveat to creating multiple replicas on a single node: make sure the replica is configured with different storage directories or different replica names!
Jon Hall
@jhall11
Feb 28 2017 19:13
@kuujo It seems in our continuous node restart test, atomix-1.0.2 is failing faster than atomix-1.0.1. I have some logs from out last run. Still digging into it, but it looks like some application queries on node three timeouts are timing out. I’m not actually seeing the client submitting the queries, but I could have missed something
Jordan Halterman
@kuujo
Feb 28 2017 19:54
@jhall11 the logs are not corrupt, so that's one important part. But what I do see on node 2 is at some point commands still stop being completed for some reason. They're logged and replicated and committed and even applied (which is beyond the issue that was occurring before) but I just don't see a CommandResponse sent back to the client.
Jon Hall
@jhall11
Feb 28 2017 19:55
hmm
vishwas
@vishwass
Feb 28 2017 20:14
@kuujo I want to change the data structure. and replace the logs
with other format
Jordan Halterman
@kuujo
Feb 28 2017 20:22
@jhall11 FYI I’m looking at session 32363 in partition 2 on node 2. That node becomes the leader and then sends a PublishRequest to the session and starts getting commands. I already checked the PublishRequest and all the sequence numbers and they’re all in the state necessary to allow the session to progress. But after node 2 becomes leader for partition 2, it starts getting CommandRequest from that session, logs CommandEntry, sends AppendRequest, commits and then applies CommandEntry, but never sends a CommandResponse.
e.g.
ClientSession            32363 - Sending CommandRequest[session=32363, sequence=112, command=InstanceCommand[resource=54, command=ResourceCommand[command=Withdraw{topic=work-partition-2}]]]
LeaderState              172.17.0.3:9876 - Received CommandRequest[session=32363, sequence=112, command=InstanceCommand[resource=54, command=ResourceCommand[command=Withdraw{topic=work-partition-2}]]]
LeaderState              172.17.0.3:9876 - Appended CommandEntry[index=35691, term=19, session=32363, sequence=112, timestamp=1488269331785, command=InstanceCommand[resource=54, command=ResourceCommand[command=Withdraw{topic=work-partition-2}]]]
LeaderAppender           172.17.0.3:9876 - Sent AppendRequest[term=19, leader=-1408161302, logIndex=35690, logTerm=19, entries=[1], commitIndex=35690, globalIndex=35680] to /172.17.0.4:9876
LeaderAppender           172.17.0.3:9876 - Received AppendResponse[status=OK, error=null, term=19, succeeded=true, logIndex=35691] from /172.17.0.4:9876
ServerStateMachine       172.17.0.3:9876 - Applying CommandEntry[index=35691, term=19, session=32363, sequence=112, timestamp=1488269331785, command=InstanceCommand[resource=54, command=ResourceCommand[command=Withdraw{topic=work-partition-2}]]]
there’s never any Sent CommandResponse[…]
the session’s previous queries with sequence=111 completed, indicating that the state in the server is correct and sequence=112 should have immediately been applied
@vishwass all you have to do is modify Log, but it’s not really just as simple as just appending to a file
well, not if you want the cluster to work anyways
but here’s Log
Jordan Halterman
@kuujo
Feb 28 2017 20:27
really, if you just want to see what’s written to the log in human readable text, you should just enable DEBUG logging, which shows all of that information
So @jhall11 that command gets to apply(Entry)
Which calls apply(CommandEntry)
Which since it seems to be sequenced correctly should get to executeCommand which should have applied it to the state machine and then completed the future so...
vishwas
@vishwass
Feb 28 2017 20:32
you had told me prevoiusly told me
to use openlog for viewing binary files
should we call this in the terminal ?
Jordan Halterman
@kuujo
Feb 28 2017 20:34
You'd have to write your own code to do it. Create a Storage object like you would when creating a CopycatServer. Point it to the appropriate log directory, and call openLog("copycat") on the Storage object. That will allow you to read the log programmatically and you can do whatever you want with that information.
You just have to configure the serializer to read whatever entries are in there
vishwas
@vishwass
Feb 28 2017 20:35
ok got
i
it
Jordan Halterman
@kuujo
Feb 28 2017 20:36
Copycat 2.0 will have some utilities to read the logs, but it's not possible right now because of serialization
@jhall11 I think we need to put some try-catch blocks inside the ServerStateMachine apply method and print the stack trace. I'm guessing that a different exception that's not being handled and is just disappearing. There doesn't seem to be any other reason the commands are not competed
completed*
Jordan Halterman
@kuujo
Feb 28 2017 20:45
@jhall11 atomix/copycat#292
from other sessions I’m able to deduce that commands are being applied successfully, but not completed for some reason. When a command is applied, the sequence number is being incremented...
ServerStateMachine       172.17.0.3:9876 - Applying CommandEntry[index=35698, term=19, session=35694, sequence=3, timestamp=1488269333078, command=io.atomix.manager.internal.GetResource@42cd6dd3]
LeaderState              172.17.0.3:9876 - Received QueryRequest[session=35694, sequence=3, index=35698, query=InstanceQuery[resource=7, query=ResourceQuery[query=io.atomix.resource.internal.ResourceQuery$Config@7120f5d1], consistency=null]]
ServerStateMachine       172.17.0.3:9876 - Applying QueryEntry[index=35698, term=19, session=35694, sequence=3, timestamp=1488269333082, query=InstanceQuery[resource=7, query=ResourceQuery[query=io.atomix.resource.internal.ResourceQuery$Config@7120f5d1], consistency=null]]
LeaderAppender           172.17.0.3:9876 - Sent AppendRequest[term=19, leader=-1408161302, logIndex=35698, logTerm=19, entries=[0], commitIndex=35698, globalIndex=35680] to /172.17.0.4:9876
LeaderAppender           172.17.0.3:9876 - Received AppendResponse[status=OK, error=null, term=19, succeeded=true, logIndex=35698] from /172.17.0.4:9876
LeaderState              172.17.0.3:9876 - Sent QueryResponse[status=OK, error=null, index=35698, eventIndex=35694, result={}]
the leader applies a command and then a query from the same session. A response is sent for the query but not for the command
that means the command did indeed make it into executeCommand
Jordan Halterman
@kuujo
Feb 28 2017 20:51
how the future is not then completed and the response is not sent I have no idea
it has to be inside that method, because commands and queries use the same completeOperation method after that to create a response
one thing I’ve seen in Copycat 2.0 testing is an NPE on this line: https://github.com/atomix/copycat/blob/master/server/src/main/java/io/atomix/copycat/server/state/PassiveState.java#L373 but I haven’t figured out why that is
the command future is always completed with a non-null result if it’s not failed
vishwas
@vishwass
Feb 28 2017 20:55
@kuujo how to check the logs are being replicated in all the server
Jordan Halterman
@kuujo
Feb 28 2017 20:55
maybe I’m missing something
@vishwass enable DEBUG logging
vishwas
@vishwass
Feb 28 2017 20:56
ok thanks :)
Jordan Halterman
@kuujo
Feb 28 2017 20:56
change this line to DEBUG if you’re using the example server
it will show every time an entry is appended to the log, committed, applied to the state machine, etc
then just watch the logs on all the servers
you’ll see the leader appending to its log and replicating to followers
vishwas
@vishwass
Feb 28 2017 20:57
thats cool
Jon Hall
@jhall11
Feb 28 2017 21:26
@kuujo, Hmm, that does indeed seem strange. I will retry and see if we catch anything
vishwas
@vishwass
Feb 28 2017 21:33
@kuujo
now the logs are working
Jordan Halterman
@kuujo
Feb 28 2017 21:59
The good news is that the logs seem to show that the two bugs that were fixed are not the problem. I really wonder if it's the same NPE I've seen in the 2.0 branch. I still haven't figured out why it happens. I don't see much reason for executeCommand to throw any other exceptions that aren't caught.
Jon Hall
@jhall11
Feb 28 2017 21:59
thats good :)
on a somewhat related note, we saw some performance improvements in 1.0.1 that went away in 1.0.2. We are investigating to see if those were anomalies or not
Jordan Halterman
@kuujo
Feb 28 2017 22:02
The only performance related change was an improvement in performance when a follower was down that was pretty noticeable from my end: atomix/copycat#285
Could also be a consequence of which node a client ends up connected to for most of a test
Jon Hall
@jhall11
Feb 28 2017 22:05
yeah, we are gonna try to figure out what happened and if we can optimize for that. I think the difference is how long it takes the last node to receive the event vs. the first node
Jordan Halterman
@kuujo
Feb 28 2017 23:09

@jhall11 Yeah, so on event-heavy resources in a blocking API it can be expensive to switch between events sent to the client and operations sent by the client. If the client sends a command to a follower which gets proxied to the leader then it will get a response back immediately. But an event will have to be replicated to the follower and committed and applied on a later heartbeat before it's sent to the client. Since the client won't complete the next command or query until that event is received (for FIFO consistency on the client) it can block later commands/queries until another round trip in the cluster. That basically makes the second command block for two round trips after the first, instead of one. If the follower to which the client is connected didn't participate in the commit, it could be more. That can be fixed by connecting the client to the leader with ServerSelectionStrategies.LEADER. IIRC right now it's ANY. We can also improve the protocol to ensure the client connects to a server that's actually participating in the commit (it's log contains commitIndex)

Any minor performance issues will also be easily overcome by the new log though :-)

Jon Hall
@jhall11
Feb 28 2017 23:12
that make sense
Jordan Halterman
@kuujo
Feb 28 2017 23:16
It only really makes sense to connect to followers if events aren't heavily used and SEQUENTIAL reads are. The leader still ultimately has the same overhead from operations being proxied. And the client will still connect to followers if it has to using the LEADER strategy so it shouldn't cause any issues
Jon Hall
@jhall11
Feb 28 2017 23:29
I guess it depends on what type of operations you want to optimize for. Some of the distributed primitives we use are SEQUENTIAL and some are LINEARALIZABLE.
Jordan Halterman
@kuujo
Feb 28 2017 23:34
Hrms...