I'm pleased to announce that the latest eventuate-0.5-SNAPSHOT now passes all chaos test scenarios
of the eventuate-chaos
project. For a high-level description, see for example the operation-based Counter CRDT test scenario
which was also executed with a Cassandra storage backend
. Chaos was generated by randomly injecting network partitions between Eventuate locations, within the Cassandra cluster and between Eventuate locations and the Cassandra cluster nodes. Furthermore, network packets have been dropped randomly too. In all tests, replicated state has been updated concurrently at rates between 10 - 100 events per second per location. This was a reasonable update rate given the high frequency of network partitions and network packet loss. After a final annealing phase, replicated state successfully converged at all locations, even after long-running chaos tests (approx. 1 hour). A single missing event or a duplicate would have broken convergence and let the chaos tests fail. The failure handling consequences learned during the chaos tests are now summarized in the failure handling
section of the reference documentation
. We will later provide a more detailed chaos test report in a separate blog post.