These are chat archives for atomix/atomix

23rd
Apr 2017
sturcotte06
@sturcotte06
Apr 23 2017 00:05
Also, I'm trying to submit multiple operations that imply disk writes (distributed file system kind of implementation) through 3 local copycat servers, but the servers and the clients keep disconnecting. The client prints multiple times "Failed to connect to the cluster" (like 100 times in the log each 5sec or so). Is there a way to make sure the cluster stays alive even on IO operations? Is it possible to partition operations by path? Like, a guaranteed order of operations, but only on opoerations of the same key.
yashhema
@yashhema
Apr 23 2017 01:06
Hello, I am trying to learn about raft. As a learners project, I want to use raft consensus to replicate data (some files) and maintain state across the members ie which server contains primary files and which contains replicated set . I am trying to understand which components I require : CopyCat or Atomix or both.
sturcotte06
@sturcotte06
Apr 23 2017 01:16
@bhardwajrajesh Hello, I'm trying to do the same thing, and I'm slowly trying to understand how everything works
You first need to abstract your storage, basically, you need to split your storage into multiple design patterns applied one after the other. Replication is one of these patterns
CopyCat can be used, but I've has a hard time with the actual data streams
however, copy cat guarantees the order of events, so if you can deal with the streaming, you can guarantee that commands applied to your files are applied in order
and as part of your command commit, you can retrieve your data (via another socket for example), and complete the write command
Jordan Halterman
@kuujo
Apr 23 2017 09:01
Apparently Gitter just likes to lose long messages. I wrote one that seems to have gotten lost. I'll have to try again in the morning.
Sometimes they magically reappear though, and I don't want to rewrite all that :-P
Jordan Halterman
@kuujo
Apr 23 2017 09:44

@sturcotte06 I'll give you the short version and we can just discuss it...

If you're trying to write files inside of a state machine, IMO you're doing something wrong. It's important not to think about Copycat or consensus protocols as replication protocols. They're best for coordination. Streaming is completely antithetical to the guarantees made by Copycat. It guarantees that operations are atomic. Streams are infinite. Operations in Copycat/Atomix must be small. Sure, you could break a stream into a bunch of smaller commits, but that would actually eliminate many of the most important guarantees. And sure, you can partition the cluster (just create multiple clusters), but it still won't behave the way you want it to. Copycat clients aggressively handle failures by retrying commands, which is what you're seeing in the logs. Expensive I/O is interpreted as failures by clients. And Copycat is currently not well suited to persistent state machines (writing to disk in a state machine). And if you don't do it correctly (manage indexes on disk), you will lose state (after a leader compacts its logs).

Consensus is really a bad protocol for handling large amounts of data in general. A distributed file system should be using a consensus algorithm to store metadata and coordinate the cluster, and it should likely be using primary/backup replication for the files themselves. The problem is, consensus forces you to use larger disk space, which becomes a finite resource when you're building a file system.

Using Copycat or any consensus protocol to store files in a file system is the wrong way to think about consensus. You should be using it to coordinate the file system, and use more efficient protocols to store the files themselves. This is why Atomix provides primitives for coordinating clusters, and this is precisely how we use consensus and Copycat/Atomix. We have a partitioned Atomix cluster that manages state machines for various primitives that store configuration information and do coordination like leader election, transactions, and distributed locking. But applications use those primitives externally to share information with one another.

If I were building a distributed file system, I'd use consensus to store critical state like file system paths and file locations, and use a primary-backup replication protocol to store the files themselves. I'd use leader election or distribute transactions to manage updates to file system metadata, and use distributed locks to manage concurrent access to files.

sturcotte06
@sturcotte06
Apr 23 2017 13:56
This makes sense. Would atomix be able to handle millions of distributed locks? In fact, I'm trying to implement an object storage that will contain million of small objects. During the course of execution (long running process), the storage will have operations on millions of those objects. If each write operation requires a lock, then I might end up with a lot of distributed lock creation. Would it be a problem on atomix' side?
sturcotte06
@sturcotte06
Apr 23 2017 14:12
Also, when trying to create a DistributedLock, I get: 2017-04-23 10:10:46,214 ERROR io.atomix.catalyst.concurrent.SingleThreadContext An uncaught exception occurred
java.lang.IndexOutOfBoundsException: inconsistent index: 1
at io.atomix.catalyst.util.Assert.index(Assert.java:45)
at io.atomix.copycat.server.storage.Segment.append(Segment.java:336)
at io.atomix.copycat.server.storage.Log.append(Log.java:294)
at io.atomix.copycat.server.state.ActiveState.appendEntries(ActiveState.java:107)
sturcotte06
@sturcotte06
Apr 23 2017 14:33
Also got this one:
java.lang.IllegalStateException: failed to initialize Raft state
at io.atomix.copycat.server.state.ServerContext.transition(ServerContext.java:612)
at io.atomix.copycat.server.state.FollowerState.sendPollRequests(FollowerState.java:107)
at io.atomix.copycat.server.state.FollowerState.lambda$resetHeartbeatTimeout$1(FollowerState.java:86)
at io.atomix.catalyst.concurrent.Runnables.lambda$logFailure$0(Runnables.java:20)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Initialize entry not appended at the start of the leader's term
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at io.atomix.copycat.server.state.ServerContext.transition(ServerContext.java:610)
... 10 more
Caused by: java.lang.IllegalStateException: Initialize entry not appended at the start of the leader's term
at io.atomix.catalyst.util.Assert.state(Assert.java:69)
at io.atomix.copycat.server.state.LeaderState.appendInitialEntries(LeaderState.java:94)
at io.atomix.copycat.server.state.LeaderState.open(LeaderState.java:66)
at io.atomix.copycat.server.state.ServerContext.transition(ServerContext.java:610)
at io.atomix.copycat.server.state.CandidateState.sendVoteRequests(CandidateState.java:106)
at io.atomix.copycat.server.state.CandidateState.startElection(CandidateState.java:65)
at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705)
at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:717)
at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2010)
at io.atomix.copycat.server.state.CandidateState.open(CandidateState.java:57)
... 11 more
sturcotte06
@sturcotte06
Apr 23 2017 14:42
``````
Does this code make sense? The lock method always return null. Trying to call bootstrap() on the replica always fail.
public static void main(String[] args) throws InterruptedException {
        List<Address> cluster = Arrays.asList
                (new Address("127.0.0.1", 4000),
                 new Address("127.0.0.1", 4001),
                 new Address("127.0.0.1", 4002));

        List<CompletableFuture<?>> bootstrapFutures = new ArrayList<>();
        for (int i = 0; i < cluster.size(); i++) {
            bootstrapFutures.add(createReplica(cluster.get(i))
                                         .server()
                                         .bootstrap(cluster));
        }

        bootstrapFutures.forEach(CompletableFuture::join);

        AtomixClient client = AtomixClient
                .builder()
                .withTransport(new NettyTransport())
                .build();

        client.connect(cluster).join();
        client.getLock(ObjectId.of("root01", "test01").value())
              .thenAccept(DistributedLock::lock)
              .thenAccept(System.out::println);
    }

    private static AtomixReplica createReplica(Address address) {
        return AtomixReplica
                .builder(address)
                .withTransport(NettyTransport
                                       .builder()
                                       .withTcpKeepAlive()
                                       .withTcpNoDelay()
                                       .withThreads(4)
                                       .build())
                .withStorage(Storage.builder()
                                    .withStorageLevel(StorageLevel.MEMORY)
                                    .build())
                .withClusterManager(BalancingClusterManager
                                            .builder()
                                            .withQuorumHint(3)
                                            .withBackupCount(1)
                                            .build())
                .build();
    }