Thanks for suggestions though! I've learnt some new things and that's always good.
dpc It's small piece of code
Ralith just read the paper at https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf and DIY
dpc CEC in essence is like one line of code
dpc https://crates.io/crates/rdedup-cdc , if you click repository it points to https://github.com/aidanhs/rsroll
rdup (compiler errors) so I'll have to do without it. In my understanding it just prints some paths? In the rdedup readme there's an example rdup -x /dev/null "$HOME" | rdedup store home.. does this just store a list of paths?
echo hey | rdedup store home, and it seems like loading this just returns "hey". Now... how can I use rdedup to backup file hierarchies? And how do I do it incrementally? Running rdedup store home again returns an error that home already exists.
rdedup store, with a new (probably with date) name. and then rdedup will try to store that with as little data duplication as possible compared with other stored things? Is that right? So I was contemplating using tar to create an archive of everything I want to backup and feed it to rdedup. But that would perhaps compromise the performance compared to rdup? Or not? (thinking that it might obscure some otherwise duplicated data)
dpc > <@gitter_ploppz:matrix.org> Looked a bit more it the docs/example in rdedup, and I think I understand the general workings. You would pipe all data you want tobackup into rdedup store, with a new (probably with date) name. and then rdedup will try to store that with as little data duplication as possible compared with other stored things? Is that right? So I was contemplating using tar to create an archive of everything I want to backup and feed it to rdedup. But that would perhaps compromise the performance compared to rdup? Or not? (thinking that it might obscure some otherwise duplicated data)
tar should work OK, I think.
dpc You must not use compression in tar, otherwise deduplication wouldn't work.
dpc And yes, traverser is about doing something like tar or rdup do.
dpc But if you designed it well, it would work better with deduplication in rdedup
tar), and the this time it takes 2.2GB. That's rather much considering it's the exact same directory? Is tar bad then? I did it with tar -cf - /path/to/dir
dpc rdedup du reports original size of the data you stored
dpc Deduplication will happen between multiple backups of the same stuff (with minor changes).
Dump everything in one tar.
@dpc You mean, all folders I want to backup, I should put in one tar?. Hm... It's just that I have like 300GB of stuff already that I want to backup - basically my whole life. I keep updating it with e.g. pictures from phone. And then I was thinking about having some "recent and relevant" store that is a bit smaller and keeps getting updated. This is turning into general backup advice :P