oetiker/znapzend

zfs backup with remote capabilities and mbuffer integration.

Tobias Oetiker

@oetiker

there is no such document, but basically you can parse syslog output

felixn-dk

@felixn-dk

Hi I get this error when I try to modify and list : zLog must be specified at creation time

stellarpower

@stellarpower

Having the same. Is this because it can't connect to syslogd?

Carsten Härle

@chaerle

What does the znapzend 20.0 error message mean?

cannot send 'xxxxxx': I/O error
ERROR: cannot send snapshots to xxxxx on yyyy
send/receive for data1 failed: Can't call method "message" without a package or object reference at /usr/lib/x86_64-linux-gnu/ZnapZend.<

storkone

@storkone

Hi, in lib/ZnapZend.pm $refreshBackupPlans is called during init. That sub creates (to be replicated) zfs datasets at the destination if they don't exist. However the pre-send-command and post-send-command's (if set and not off) are also executed. I find this a bit misleading because the documentation describes a valid example (bringing a zpool online) but other use cases come to mind, like running some job against the transferred snapshot. But that like the parameter name implies, run that job only before and after zfs send do not run it after "refresh" Yes the environment variable WORKER is appended with "-refresh" to distiguish both cases but that's nowhere documented.
Please let me know whether you want the documentation updated or split the pre/post-send-command in the "refresh" and send versions. Which is a bit cleaner than checking some environment variable for a string.

Tobias Oetiker

@oetiker

if you have an idea on how to make this cleaner without breaking existing implementations ...

otherwise improvements for PR the documentation will be fine

storkone

@storkone

Now a design question. Why does the pre/post-snap-cmd includes the snapshot name in an environment variable for the scripts and the pre/post-send-cmd doesn't?
Now the script doesn't know which snapshot was send. I can think of various (edge) cases where something might have changed between the actual send and executing the script. I've hacked this variable in the various sub's to get it from $createSnapshot via the workers to $sendRecvCleanup. If interested I could create a pull request.
I need post processing of the send snapshot, not a snapshot. Another snapshot might have been created, this one might have been deleted. So just processing the last snapshot won't do it. Processing get's done in [dataset]/.zfs/snapshot/[snapshot name]

@oetiker , unfortunately I don't. But I was hoping that you might see a solution.

Tobias Oetiker

@oetiker

I see no problem in providing additional information via environment variables, so please proved a PR!

Kevin Liu

@pot8to

Hey, is there any support in znapzend for deleting local snapshots while keeping remote ones? I'm worried about disks filling up with old snapshots.

Tobias Oetiker

@oetiker

that is the whole point of znapzend ... you configure exactly howmany snapshots you want to keep ... local and remote

Manuel Oetiker

@moetiker

Every location has his own plan ....

src = fast/home-a
src_plan = 1day=>30minutes,7days=>2hours,30days=>4hours

Kevin Liu

@pot8to

yup, thanks!

Carsten Härle

@chaerle

I try to send an unencrypted hierarchy to an encrypted one. The znapzend daemon fails with an error, that it cannot destroy the transfer root encrypted File system with "zfs receive -F" and terminates. However with --runonce znapzend continues with the sub datasets and transfers it correctly. It there a means to have the daemon work also?

Jim Klimov

@jimklimov

Hey, made a PR and found that all the self-tests are different now with github flows :)

The perl 5.10 support fails to install Mojolicious (needs 5.16+) and this seems to cancel other version builds.

https://github.com/oetiker/znapzend/pull/492/checks?check_run_id=865205219

Successfully installed IO-Socket-IP-0.39
! Installing the dependencies failed: Your Perl (5.010001) is not in the range '5.016'
! Bailing out the installation for Mojolicious-8.56.
! Installing the dependencies failed: Module 'Mojolicious' is not installed
! Bailing out the installation for Mojo-IOLoop-ForkCall-0.20.
! Installing the dependencies failed: Module 'Mojo::IOLoop::ForkCall' is not installed, Module 'Mojolicious' is not installed
! Bailing out the installation for /home/runner/work/znapzend/znapzend/.
Installing modules failed
21 distributions installed
Makefile:426: recipe for target 'cpanfile-5.10.1.snapshot' failed

Jim Klimov

@jimklimov

UPDATE: Hopefully fixed as part of my PR https://github.com/oetiker/znapzend/pull/492/ :)

Jim Klimov

@jimklimov

Wasn't in this channel for a while... I saw discussion above about holds or bookmarks on datasets; my take on this was that we could also portably use dataset properties (like marking certain dataset as being the last known common point of this source and that destination) and so avoid autoremoval on either side

Jim Klimov

@jimklimov

Docker image tests fail in another PR:

ERROR: unsatisfiable constraints:
perl-5.30.3-r0:
breaks: world[perl=5.30.1-r0]
satisfies: autoconf-2.69-r2[perl]
automake-1.16.1-r0[perl]

maybe some new alpine release is being rolled out and this got fetching in-between?..

Jim Klimov

@jimklimov

@Rivqua > I have a question I can't figure out the answer to, I've setup znapzend, and it works. I am wondering though, how do I configure the features? like --features=skipIntermediates,compressed ?

This usually happens on the command line, either for the standalone CLI tool like znapzend --runonce=pool/export --features=... or similarly for the service definition. Now in the multi-distro world, the specifics of passing your custom CLI arguments to the service (systemd? SMF? init-script?..) would differ, but the idea remains.

Jim Klimov

@jimklimov

@oetiker : I'm getting lost in trying to cheat around Test::More and family (for where daemonized mode is tested)

whatever I try, there are some modes of invocation that succeed and some that fail, no silver bullet yet :(

Tobias Oetiker

@oetiker

keep fighting :)

David Česal

@Dacesilian

Hello, when I type "znapzendzetup list", it says only "zLog must be specified at creation time!". How can I fix this, please?

Tobias Oetiker

@oetiker

this is fixed in master

David Česal

@Dacesilian

@oetiker Can you please write more detailed tutorial how to build it?

git clone https://github.com/oetiker/znapzend.git znapzend
cd znapzend/
autoconf
aclocal
./configure --prefix=/opt/znapzend

Can't open perl script "/root/znapzend/thirdparty/carton/bin/carton": No such file or directory

David Česal

@Dacesilian

In Debian
apt install carton
make -j4

Tobias Oetiker

@oetiker

hmmm run ./bootstrap

then configure make

David Česal

@Dacesilian

Okay, this is working fine, thanks. Important is to use only one-threaded make (not make -j4) to let carton download all dependencies.

git clone https://github.com/oetiker/znapzend.git znapzend
cd znapzend/
./bootstrap.sh
./configure --prefix=/opt/znapzend-master
make
make install
rm /usr/local/bin/znapzend
for x in /opt/znapzend-master/bin/; do ln -s $x /usr/local/bin; done

@oetiker But error is the same, what can I do?
/opt/znapzend-master/bin# ./znapzendzetup list
zLog must be specified at creation time!

Tobias Oetiker

@oetiker

are you calling list without having anything setup ? then this is just a bad error message ...

David Česal

@Dacesilian

@oetiker No, I have setup done, backup is working. But from some point of time, znapzendzetup list does not work. On my other server, it's working fine. :\

@oetiker Maybe it's connected with fact that I've excluded some child dataset. Hmm, znapzend should be able to handle this situation, I guess.

David Česal

@Dacesilian

I'm not sure if excluded dataset is on this server. Anyway, it's working when I specify pool name: znapzendzetup list --recursive nvme

Tobias Oetiker

@oetiker

@Dacesilian can you try this patch:

diff --git a/lib/ZnapZend/Config.pm b/lib/ZnapZend/Config.pm
index 6bc4249..c616729 100644
--- a/lib/ZnapZend/Config.pm
+++ b/lib/ZnapZend/Config.pm
@@ -33,7 +33,8 @@ has zfs  => sub {
         rootExec => $self->rootExec,
         debug => $self->debug,
         lowmemRecurse => $self->lowmemRecurse,
-        zfsGetType => $self->zfsGetType
+        zfsGetType => $self->zfsGetType,
+        zLog => $self->zLog
     );
 };
 has time => sub { ZnapZend::Time->new(timeWarp=>shift->timeWarp); };

Jim Klimov

@jimklimov

I think the fix is in recent commits for one of those two PRs in review, as well. Side effect of going away from raw warn() noticed late :(

Needed to pass a trivial new Mojo::Log object so these tools spam formatted logs to stderr

David Česal

@Dacesilian

Thank you, I've compiled current master branch and it is working fine!

rm -r znapzend/
git clone https://github.com/oetiker/znapzend.git znapzend && cd znapzend/

apt-get install perl unzip autoconf carton
./bootstrap.sh
./configure --prefix=/opt/znapzend-master
make
make install
rm /usr/local/bin/znapzend*
for x in /opt/znapzend-master/bin/*; do ln -s $x /usr/local/bin; done

Jim Klimov

@jimklimov

Note: I'm just updating the README in https://github.com/oetiker/znapzend/pull/512/ so thanks for the updated apt-get bit ;)

As for the symlinks, a ln -f takes care of removing older ones if present, and a GNU ln -r can make relative symlinks that are more meaningful when you juggle many alternate roots and do not want to reference the current OS you are running (most linux systems/scripts do not know the difference... solaris had that for decades, e.g. a file server hosting roots for diskless NFS workstations, so it is sort of a built-in habit)

unfortunately a non-GNU ln such as in Solaris does not have the -r so explicit ../ have to be prepended

Jim Klimov

@jimklimov

Thanks for confirming the rest works well in the isolated system :)

Jim Klimov

@jimklimov

@oetiker : is there something remaining to be resolved urgently in https://github.com/oetiker/znapzend/pull/497/ or can you merge it? :)

Jim Klimov

@jimklimov

And also oetiker/znapzend#508 and oetiker/znapzend#512

Jim Klimov

@jimklimov

and oetiker/znapzend#515 :)

Where communities thrive

People

Repo info

Activity