Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    tionis
    @tionis:matrix.org
    [m]
    Ok, so if I want to read a specific file from snapshots I should ensure that it is one of the lexicographic first for faster lookup?
    andrewchambers
    @andrewchambers
    How many files do you plan to put in each snapshot?
    1 reply
    andrewchambers
    @andrewchambers
    In the future I might improve bupstash to do something more efficient in that case
    but I can't really say without knowing what you are doing
    andrewchambers
    @andrewchambers
    @tionis:matrix.org btw - there is no sponsor page, the best way would be to create an empty repository on bupstash.io
    it would just charge you the minimum
    tionis
    @tionis:matrix.org
    [m]
    Ok
    andrewchambers
    @andrewchambers
    So, a new release is coming soon, it is almost time for 1.0 or at least 1.0 release candidates
    1 reply
    1.0 will mean even stronger backwards compatibility guarantees - not that I ever deliberately broke a repository
    1.0 should coincide with wider access to the bupstash.io service - so we really don't want to break bupstash for people using versions packaged with their distro package manager.
    andrewchambers
    @andrewchambers
    Though we still need a logo / fav icon :P
    tionis
    @tionis:matrix.org
    [m]
    Is there some way to force bupstash new-key to print the key to stdout instead of a file
    tionis
    @tionis:matrix.org
    [m]
    :point_up: Edit: Is there some way to force bupstash new-key to print the key to stdout instead of a file?
    andrewchambers
    @andrewchambers
    it seems not
    we couldatt it I suppose
    add
    andrewchambers
    @andrewchambers
    The main reason it doesn't is it sets the permissions on the key
    andrewchambers
    @andrewchambers
    so you don't accidentally make a key readable by other users, but stdout with -o - might be ok
    tionis
    @tionis:matrix.org
    [m]
    Would make it a tiny bit easier to wrap bupstash in scripts to store the key in a secure place like the keychain
    piegames
    @piegames:matrix.org
    [m]
    Does using /dev/stdout as "file" work?
    Paul
    @ptman:kapsi.fi
    [m]
    isn't that like a bashism?
    oh, it isn't?
    symlinks to /proc/self/fd/1 (and 0 and 2)
    maybe I was thinking of bash /dev/tcp
    andrewchambers
    @andrewchambers
    It currently doesn't work since it doesn't overwrite keys
    tionis
    @tionis:matrix.org
    [m]
    Does bupstash currently do checksumming etc on the server side to prevent corruption of data?
    andrewchambers
    @andrewchambers
    @tionis:matrix.org all data is hashed and signed so if it got corrupted on the server
    the client would alert you
    however if you want checksums on your repository data too, I considered adding more, but I would also say you could use the btrfs or zfs file systems
    those filesystems verify checksums on read automatically for all data
    andrewchambers
    @andrewchambers
    for my personal backups I use btrfs so I can use btrfs scrub
    or I can just read the data
    and it will check for errors
    andrewchambers
    @andrewchambers
    That being said, in the future I might add a data storage mode that adds parity information and checksums as an extra
    for now I would advise using a filesystem with checks or RAID
    tionis
    @tionis:matrix.org
    [m]
    Ok, thanks
    nh2
    @nh2:matrix.org
    [m]

    @andrewchambers: can you remind me, what exactly is bupstash's behaviour with small files, and does it depend on if they are in the same dir or not?
    In https://github.com/andrewchambers/bupstash/issues/26#issuecomment-730882871 you write

    small files are packed into single chunks

    but that is the only reference I can find on the Internet for this feature

    andrewchambers
    @andrewchambers
    @nh2:matrix.org sure
    so bupstash packs small files within the same directory into a chunk
    bupstash internally uses directory boundaries as a sort of deduplication boundary
    4 replies
    that helps the dedup resync
    as a consequence it means it must split the chunk at directory boundaries and isn't able to pack more small files into it
    I am considering other heuristics that can group directories
    technically it would be possible to pack many directories into a single chunk
    it doesn't affect the repository format - it is something we can change
    bupstash concatenates all files into a giant stream
    all file data into a giant stream
    and dedups that by just splitting it into chunks
    our index is another stream that stores file names and their offsets into the data stream
    it is quite different to every other backup tool I have seen