These are chat archives for atomix/atomix

28th
Aug 2018
suimi
@suimi
Aug 28 2018 04:32
@kuujo i applied this commit, it's fine
Jordan Halterman
@kuujo
Aug 28 2018 04:32
:+1: thanks!
suimi
@suimi
Aug 28 2018 04:33
i also add the debug log at flush segment
 @Override
  public void flush() {
    buffer.flush();
    log.debug("segment writer flushed");
  }
2018-08-28 12:20:59,600 [raft-server-raft-group-partition-1] DEBUG JournalSegmentWriter - segment writer flushed [,,]
2018-08-28 12:20:59,620 [raft-server-raft-group-partition-1] DEBUG SegmentedJournal - Created disk segment: JournalSegment{id=2, version=1, index=80, size=64} [,,]
the previous segment called flush, but seem not flush to disk
2018-08-28 12:23:43,563 [atomix-partition-group-membership-service-0] DEBUG SegmentedJournal - Loaded disk segment: 1 (system-partition-1-1.log) [,,]
2018-08-28 12:23:43,564 [atomix-partition-group-membership-service-0] DEBUG SegmentedJournal - Found segment: 1 (system-partition-1-1.log) [,,]
2018-08-28 12:23:44,979 [raft-client-system-partition-1-6] DEBUG SegmentedJournal - Loaded disk segment: 1 (raft-group-partition-1-1.log) [,,]
2018-08-28 12:23:44,979 [raft-client-system-partition-1-6] DEBUG SegmentedJournal - Found segment: 1 (raft-group-partition-1-1.log) [,,]
2018-08-28 12:23:44,981 [raft-client-system-partition-1-6] DEBUG SegmentedJournal - Loaded disk segment: 2 (raft-group-partition-1-2.log) [,,]
2018-08-28 12:23:44,981 [raft-client-system-partition-1-6] DEBUG SegmentedJournal - Found segment: 2 (raft-group-partition-1-2.log) [,,]
2018-08-28 12:23:44,981 [raft-client-system-partition-1-6] WARN  SegmentedJournal - Journal is inconsistent. /data/home/admin/trust/data/atomix/raft-group/2/partitions/1/raft-group-partition-1-2.log is not aligned with prior segment /data/home/admin/trust/data/atomix/raft-group/2/partitions/1/raft-group-partition-1-1.log [,,]
suimi
@suimi
Aug 28 2018 05:15
@kuujo there have other issue, if apply this commit, when the cluster stopped at the same time, cluster will be missing some log at restartting
maybe need check why the log not flushed to disk after called it
suimi
@suimi
Aug 28 2018 07:10
@kuujo i checked the javadoc of getFD().sync()
      * sync only affects buffers downstream of this FileDescriptor.  If
     * any in-memory buffering is being done by the application (for
     * example, by a BufferedOutputStream object), those buffers must
     * be flushed into the FileDescriptor (for example, by invoking
     * OutputStream.flush) before that data will be affected by sync.
maybe the memory buffer need flush() at writer
Jordan Halterman
@kuujo
Aug 28 2018 07:39
It’s a RandomAccessFile... there is no buffer
@suimi
suimi
@suimi
Aug 28 2018 07:49
@kuujo private final Buffer memory = HeapBuffer.allocate().flip();
Jordan Halterman
@kuujo
Aug 28 2018 07:50
What about it?
suimi
@suimi
Aug 28 2018 07:50
like this
    // Create a single byte[] in memory for the entire entry and write it as a batch to the underlying buffer.
    buffer.write(memory.clear()
        .writeInt(length)
        .writeUnsignedInt(checksum)
        .write(bytes)
        .flip().flush());
Jordan Halterman
@kuujo
Aug 28 2018 07:51
That’s just the buffer used to serialize bytes. It’s written to a RandomAccessFile. Flushing it does nothing
The first buffer.write(...) call transfers those bytes to the RAF
Junbo Ruan
@aruanruan
Aug 28 2018 07:53
writing log & flushing is separated ops, when crash before flush, the buffer of 'FILE' may not commit to DISK
Jordan Halterman
@kuujo
Aug 28 2018 07:54
Yeah, but what we’re seeing is missing entries even after the log is supposedly written to disk
But entries from the prior segment are still missing after the node is killed and restarted
currentWriter.flush() ultimately calls randomAccessFile.getFD().sync()
It’s strange
I’m going to rewrite the writer to use FileChannel, but I’m not optimistic it will fix this problem, which seems to be related to the OS
Johno Crawford
@johnou
Aug 28 2018 07:57
@kuujo you saw the new Files api right?
it does well in benchmarks
Jordan Halterman
@kuujo
Aug 28 2018 07:57
I can’t find benchmarks
But FileChannel is used for memory mapped files of course...
Johno Crawford
@johnou
Aug 28 2018 07:58
# Run complete. Total time: 02:15:57

Benchmark                                          (nbBytes)   Mode  Cnt       Score       Error  Units
InputStreamBenchmark.testFileChannel                      16  thrpt  200  386648.771 ±  2124.378  ops/s
InputStreamBenchmark.testFileChannel                    4096  thrpt  200  361793.938 ±  2135.226  ops/s
InputStreamBenchmark.testFileChannel                   32178  thrpt  200  132257.453 ±   695.310  ops/s
InputStreamBenchmark.testFileChannel                  500000  thrpt  200   11290.554 ±    70.235  ops/s
InputStreamBenchmark.testFileChannel                 5000000  thrpt  200    1026.416 ±    18.377  ops/s
InputStreamBenchmark.testFileChannelViaRandomFile         16  thrpt  200  285918.407 ±  3260.923  ops/s
InputStreamBenchmark.testFileChannelViaRandomFile       4096  thrpt  200  264223.310 ±  3347.654  ops/s
InputStreamBenchmark.testFileChannelViaRandomFile      32178  thrpt  200  116982.502 ±   556.400  ops/s
InputStreamBenchmark.testFileChannelViaRandomFile     500000  thrpt  200   11147.846 ±    53.195  ops/s
InputStreamBenchmark.testFileChannelViaRandomFile    5000000  thrpt  200    1100.950 ±     9.935  ops/s
InputStreamBenchmark.testFileInputStream                  16  thrpt  200  264379.376 ±  9758.624  ops/s
InputStreamBenchmark.testFileInputStream                4096  thrpt  200  248322.517 ± 10508.086  ops/s
InputStreamBenchmark.testFileInputStream               32178  thrpt  200  114791.194 ±  3853.470  ops/s
InputStreamBenchmark.testFileInputStream              500000  thrpt  200   10547.612 ±   221.885  ops/s
InputStreamBenchmark.testFileInputStream             5000000  thrpt  200    1112.075 ±    10.339  ops/s
InputStreamBenchmark.testFiles                            16  thrpt  200  389879.178 ±  2083.221  ops/s
InputStreamBenchmark.testFiles                          4096  thrpt  200  359445.473 ±  2302.846  ops/s
InputStreamBenchmark.testFiles                         32178  thrpt  200  130958.544 ±   578.130  ops/s
InputStreamBenchmark.testFiles                        500000  thrpt  200   11346.452 ±    54.318  ops/s
InputStreamBenchmark.testFiles                       5000000  thrpt  200    1088.265 ±    10.730  ops/s
from what I can see files is within the error margin of filechannel
Jordan Halterman
@kuujo
Aug 28 2018 08:02
But these are read benchmarks. I care about writes
Johno Crawford
@johnou
Aug 28 2018 08:02
could adapt the jmh for writes?
Jordan Halterman
@kuujo
Aug 28 2018 08:03
Maybe tomorrow... I gotta take a nap
Junbo Ruan
@aruanruan
Aug 28 2018 08:04
we can not avoid crashing one node, the raft can fix the log when it restarting from other nodes
what we do is keeping integrity of the logs written
Jordan Halterman
@kuujo
Aug 28 2018 08:09
The problem is not recovering a crashed node, it’s sync()ing changes to disk. A correct Raft implementation ensures at least that entries that are committed are written to physical storage such that when a majority of nodes is killed I’m utter changes are still not lost. That’s not what’s happening when getFD().sync() is used here
*committed changes
Also, because the prior segment cannot be written to disk, all the entries from the next segment are effectively lost as well. If you kill 2/3 nodes right after they roll over to a new segment and then restart them, changes are the node that wasn’t killed will get elected leader. I can easily describe a scenario where that leads to committed changes being lost.
chances*
I rarely type anything on this iPad
Jordan Halterman
@kuujo
Aug 28 2018 08:14
We don’t actually flush changes to disk in production, but when desirable Atomix should be able to be tuned to do so for the reasons above. That’s not working correctly in some environments ATM
FKrack
@FKrack
Aug 28 2018 08:32

I have updated the Atomix-Agent from 3.0.0-rc11 to 3.0.1 and I get the following exception:

Exception in thread "main" io.atomix.utils.config.ConfigurationException: Unknown properties present in configuration: managementGroup.dataDirectory
        at io.atomix.core.utils.config.PolymorphicConfigMapper.checkRemainingProperties(PolymorphicConfigMapper.java:100)
        at io.atomix.utils.config.ConfigMapper.map(ConfigMapper.java:174)
        at io.atomix.utils.config.ConfigMapper.getValue(ConfigMapper.java:292)
        at io.atomix.utils.config.ConfigMapper.mapSetters(ConfigMapper.java:208)
        at io.atomix.utils.config.ConfigMapper.map(ConfigMapper.java:169)
        at io.atomix.utils.config.ConfigMapper.map(ConfigMapper.java:132)
        at io.atomix.utils.config.ConfigMapper.loadFiles(ConfigMapper.java:90)
        at io.atomix.core.Atomix.config(Atomix.java:272)
        at io.atomix.core.Atomix.config(Atomix.java:251)
        at io.atomix.core.Atomix.config(Atomix.java:237)
        at io.atomix.agent.AtomixAgent.main(AtomixAgent.java:144)

I did not change the configuration file:

managementGroup {
  type: raft
  name: system
  partitions: 1
  data-directory: data/clusterNode_3
  members: [clusterNode_1,clusterNode_2,clusterNode_3]
}
Jordan Halterman
@kuujo
Aug 28 2018 08:42
The final release changed... I think it’s storage.directory now
All future releases since the final will be backwards compatible
FKrack
@FKrack
Aug 28 2018 08:46
Thank you for your answer. Where do I have to configure the storage.directory? Is it in the managementGroup section?
Jordan Halterman
@kuujo
Aug 28 2018 08:46
Yep
Here’s a link to an example...
That one doesn’t have a directory but it’s the same place as storage.level
That one has it
FKrack
@FKrack
Aug 28 2018 08:51
That does not work on my side. I get the same exception with storage now:
Exception in thread "main" io.atomix.utils.config.ConfigurationException: Unknown properties present in configuration: storage
        at io.atomix.core.utils.config.PolymorphicConfigMapper.checkRemainingProperties(PolymorphicConfigMapper.java:100)
        at io.atomix.utils.config.ConfigMapper.map(ConfigMapper.java:174)
        at io.atomix.utils.config.ConfigMapper.map(ConfigMapper.java:132)
        at io.atomix.utils.config.ConfigMapper.loadFiles(ConfigMapper.java:90)
        at io.atomix.core.Atomix.config(Atomix.java:272)
        at io.atomix.core.Atomix.config(Atomix.java:251)
        at io.atomix.core.Atomix.config(Atomix.java:237)
        at io.atomix.agent.AtomixAgent.main(AtomixAgent.java:144)
Jordan Halterman
@kuujo
Aug 28 2018 08:53
Paste the managementGroup section
Strange
It’s used in unit tests and the test framework
Wait that error seems to indicate it’s not under management-group
Look at the links
FKrack
@FKrack
Aug 28 2018 09:03
This is my managementGroup
managementGroup {
  type: raft
  name: system
  partitions: 1
  storage.directory: data/clusterNode_3
  members: [clusterNode_1,clusterNode_2,clusterNode_3]
}
Sorry! You are right! I have added a second section behind! It works!
I am so sorry! I missed the section! Thank you for your help!