These are chat archives for atomix/atomix
We have to rely on the user to release the TTL commit after the time expires and release all prior related commits before the TTL commit. In order to ensure a TTL commit is not removed from disk before prior commits, we compact the entire log sequentially.
Copycat 2.0/atomix-raft eliminates the incremental compaction algorithm altogether. The optimizations it allows for are amazing. Incremental compaction has allowed a lot of entries to be excluded from replication in ONOS. But it's a trade off. These types of complexities risk too many bugs in production systems IMO, and performance can be (and has been) regained by simplifying the algorithm and the overhead of managing logs.
Instead of incremental compaction, since Atomix Raft supports multiple state machines per cluster, we can do snapshotting in a more optimal way. Atomix Raft takes snapshots of individual state machines in memory and writes them to disk asynchronously in the background. This trades typically moderate amounts of memory to limit issues with snapshotting where the cluster can become blocked while snapshots are written to disk. It also means much, much simpler log management for state machines and ensures clusters can run forever (which was a driver of Atomix 2.0).
Storage.openLog()API and iterating through indices, but that requires being on a Catalyst thread.