Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Tobias Oetiker
@oetiker
I am pretty sure we have not tested behavior with out of space situations
Adrian Gschwend
@ktk
ok that explains thanks
it's a smartos zone and there is a quota there so I ran into that
Hemanta Kumar G
@HemantaKG
How to stop the backup of a single of the datasets without disturbing the other dataset backup plans? (I set up multiple backup planes for zfs dataset of the same zpool)
Carsten John
@cjohn_system_admin_gitlab
I'm currently backing up systems via znapzend and I'm wondering howto secure the backup server against lateral movement of an attacker (nowadays ransomware attackers try to get rid of the backups first). If the primary fileserver is compromised it's an easy job for an able attacker to make sure the backups are deleted as the source needs ssh access to the critical zfs command on the destination server. Initiating the whole thing the other way round (running the daemon on the target server) would circumvent this issue. Theoretically this should be possible, but would perhaps need a complete rewrite.
gnasys
@gnasys
Maybe by doing a pull from the backup server : running znapzend on the backup server, defining the primary remote as the source and setting the local dataset as the destination. Didn't try that configuration but i see no reason why it should not work
I made the jump to 0.21 ver, since then everytime i launch a znapzend command (zetup, ztatz etc.) i get the message "Subroutine delay redefined at /opt/znapzend-0.21.0/lib/Mojo/IOLoop.pm line 68", is that something i should worry about, what does it mean ?
Tobias Oetiker
@oetiker
@cjohn_system_admin_gitlab you could add wrapper cmmand on the remote servers authorized keys file only allowing the use of zfs receive with appropriate options
when your main backup server gets compromised the remote server is still save
although I am not aware of any randsomware attacks that subverted zfs servers
I think this mostly happens in windows land
Carsten John
@cjohn_system_admin_gitlab
@oetiker, yes I also guess the usual ransomware will not address this. My concern is that this more or less a security by obscurity approach. If I limit the the asuthorized key to "zfs receive" only, how is the retention done? Snapshots need to be destroyed on the target system at some point in time, right?
Tobias Oetiker
@oetiker
this is true ... and not solved ... formsomething like this to work, the wrapper would have to be pretty smart, only allowing "legal" cleanup
eyJhb
@eyjhb:eyjhb.dk
[m]
Is there any plan to tag a new version of znapzend ? Having oetiker/znapzend#496 in a release would be nice
David Česal
@Dacesilian
Hello, I have errors in log: taking snapshot on tank/container failed: ERROR: cannot create snapshot tank/container@2021-09-19-153000 at /opt/znapzend-0.21.0/lib/ZnapZend/ZFS.pm line 339
When I try to create this snapshot, it normally works, but znapzend is failing.
Can be problem that I have one settings for whole tank/container (recursive) and then different settings for one specific dataset?
[2021-09-19 15:23:46.56695] [17078] [info] found a valid backup plan for tank/container...
[2021-09-19 15:23:46.56715] [17078] [info] found a valid backup plan for tank/container/subvol-165-disk-0...
In tank/container/subvol-165-disk-0, there are snapshots as should be. No snapshots are created in other dataset (tank/container and children).
David Česal
@Dacesilian

When I run znapzend with noaction:

WOULD # zfs snapshot tank/container/subvol-165-disk-0@2021-09-19-161500

WOULD # zfs snapshot -r tank/container@2021-09-19-161500

zfs list -H -o name -t filesystem,volume -r tank/container

Can there be a problem that at first, subvol-165-disk-0 snapshot is created and then recursive snapshot fails, because it already exists?

David Česal
@Dacesilian
I've added logging of executed command when creating snapshots and I think it's connected to multithreading - znapzend creates snapshot recursively and also for child at once.
David Česal
@Dacesilian
I've created issue for this: oetiker/znapzend#560
kevdogg
@kevdogg

Hi I'm having a problem with my zfs backups sending. There were many days that my scheduled plan didn't work, so I have a bunch of snapshots (about 4 days worth) that are not backed up. I've run the plan manually and here is the logs:

zfs list -H -o name -t snapshot -s creation -d 1 zroot/data/timemachine

ssh -o batchMode=yes -o ConnectTimeout=30 root@10.0.1.197 zfs list -H -o name -t snapshot -s creation -d 1 tank/backups/zfs_backup/arch-TM

zfs send -I 'zroot/data/timemachine@10-06-2021-12:00:00' 'zroot/data/timemachine@10-10-2021-10:30:23'|ssh -o batchMode=yes -o ConnectTimeout=30 'root@10.0.1.197' '/usr/local/bin/mbuffer -q -s 256k -W 600 -m 200M|zfs recv -F tank/backups/zfs_backup/arch-TM'

cannot receive incremental stream: dataset is busy
mbuffer: error: outputThread: error writing to <stdout> at offset 0x55120000: Broken pipe
mbuffer: warning: error during output to <stdout>: Broken pipe
warning: cannot send 'zroot/data/timemachine@10-07-2021-00:00:00': signal received
warning: cannot send 'zroot/data/timemachine@10-07-2021-06:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-07-2021-12:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-07-2021-18:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-08-2021-00:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-08-2021-06:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-08-2021-12:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-08-2021-18:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-09-2021-00:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-09-2021-06:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-09-2021-12:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-09-2021-18:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-10-2021-00:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-10-2021-06:00:00': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-10-2021-10:28:46': Broken pipe
warning: cannot send 'zroot/data/timemachine@10-10-2021-10:30:23': Broken pipe
cannot send 'zroot/data/timemachine': I/O error
[2021-10-10 10:31:06.50500] [80725] [warn] ERROR: cannot send snapshots to tank/backups/zfs_backup/arch-TM on root@10.0.1.197
[2021-10-10 10:31:06.50525] [80725] [warn] ERROR: suspending cleanup source dataset zroot/data/timemachine because 1 send task(s) failed:
[2021-10-10 10:31:06.50638] [80725] [warn] +--> ERROR: cannot send snapshots to tank/backups/zfs_backup/arch-TM on root@10.0.1.197
[2021-10-10 10:31:06.50655] [80725] [info] done with backupset zroot/data/timemachine in 43 seconds
[2021-10-10 10:31:06.50837] [80689] [debug] send/receive worker for zroot/data/timemachine done (80725)
znapzend (PID=80689) is done.

It seems to break at this step:

'/usr/local/bin/mbuffer -q -s 256k -W 600 -m 200M|zfs recv -F tank/backups/zfs_backup/arch-TM'
cannot receive incremental stream: dataset is busy

Is this a problem with the mbuffer command or something else?

kevdogg
@kevdogg
well I looked a long time at this one -- and I finally on the destination side unmounted and then mounted the zfs dataset. This seemed to clear the dataset is busy flag and then things worked. Weird stuff.
Jim Klimov
@jimklimov
@oetiker Sorry to nudge (and channel seems stale, at least in my browser?) but there are some PRs to look at ;)
Manuel Oetiker
@moetiker
does "ssh root@10.0.1.197 " work
Tobias Oetiker
@oetiker
@jimklimov I get mail for every PR :) was just waiting for them to settle down a bit :)
Jim Klimov
@jimklimov
@oetiker Sure you do, just seeing so many PRs not merged over a while and a year since last chat here got me worried ;p
@moetiker : was that an internal question? or about the PRs dealing with remote-filtering regex? ;)
also, got several PRs where I assume they've settled down so can be reviewed
the docker (alpine vs. mbuffer) is a hard nut to crack though, I'll probably drop that effort
mostly not about znapzend per se (maybe just using another base OS image would fare better?)
Tobias Oetiker
@oetiker
@jimklimov look at the closed PRs :)
Jim Klimov
@jimklimov
I see open ones, I have 6th sense :)
Jim Klimov
@jimklimov
gentle bump ;)
Tobias Oetiker
@oetiker
so you are happy with your PR now ?
Jim Klimov
@jimklimov
I think so, at least it did not require more tinkering in the past weeks that it's running (for code PRs)
Jim Klimov
@jimklimov
Thanks for merging :)
Guess what I was setting up today? MRTG for the home server... lots of ancient memories bubbled up :)
Jim Klimov
@jimklimov
FWIW, there are interesting discussions and addons that I put together (beside what I found in Proxmox packaging) maybe worth mentioning in some readme, with:
Things two decades old still available and still working... wondrous really
as for znapzend, did you use zfs allow delegations on Linux OpenZFS? Seems there are issues receiving as a non-root user into datasets with (direct or inherited) zoned=on as I mentioned in the PRs... did anyone by chance find a workaround other than disabling the attribute as soon as received (and probably inheriting/defining local mountpoints on the backup server to override received ones)?
Tobias Oetiker
@oetiker
No, we have only a very small linux base ... mostly omnios for all big storage boxes
if you find stuff you think would be worth adding to mrtg, please PR
Jim Klimov
@jimklimov
sure, but so far it just seems to work - just gotta read up or remember how some stuff is set up ;)
one think (still) missing is rrdcached support for "14all.cgi" rendering, and generally hints how to save rrd databases with that daemon into location mrtg would see - does it tell the files for daemon to write over unix/tcp protocol? does the daemon generally have rights to do so, or should a special instance be spun up just for mrtg? (a copy of daemon seems to be part of Proxmox self-monitoring)
also I sort of miss an ability to render several time series onto same graph even where data has same range, multiplier, etc. (e.g. all MoBo/CPU/GPU/HDD/NVMe voltages or fan speeds in one graph rather than a dozen)
and I think I can't tell MRTG to directly read a file (/sys/... or /proc/...) as a data source without forking for each non-SNMP read, right?
Jim Klimov
@jimklimov
otherwise, it is fun to see when NUT CI farm gets a PR to build, and my PC acting as one of build agents has numbers (LA, temp, fan) hop up - and UPS load too, by some 1.5 times - and then calm down... now I'm not just "hearing things" when PRs come :)
and it is impressive how things throttle nowadays... it idles doing home stuff at room temperature, then spikes to 60-70 degC for half an hour, whirring more noticeably than usual, then cools back... and how voltages change too :)
Jim Klimov
@jimklimov
Hm, in some MRTG graphs I see "cur" values reported with different non-trivial readings; in others I see zeroes all the time (while historic data, max and avg exist). Is this impacted by some arcane setting? :)
Speaking of improvement ideas, nowadays an option for SVG graphs might be reasonable... complete with crispier zooming and value reports on mouse hovering... idea impressed by Jenkins Warnings-NG reports, e.g. at https://ci.networkupstools.org/job/nut/job/nut/job/master/ right (may take time to load, host is busy and slow)