by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Tobias Oetiker
@oetiker
yes something along these lines ... but the points you listed are somethinf which prevented us from takeing that route
James Crocker
@james-crocker
And just FYI I did put that json into a zfs property to see what it'd look like... The wc -c character count on that json is 1193 - So, ZoL property values can apparently exceed 1024 char. But, as discussed challenging to read on a zfs get all - and throws off the output a bit too. The ZoL zfs property name is constrained to 256 chars.
James Crocker
@james-crocker

So, the 256 char allowed for zfs property name is quite expansive. What if - a pseudo like json zfs property? Something like

org.znapzend:source:enabled=true
org.znapzend:source:snapshot:plan=1hours=>15minutes,1days=>1hours,1weeks=>1days,1months=>1weeks,1years=>1months
org.znapzend:source:snapshot:tsformat=%Y-%m-%dT%H:%M:%SZ
org.znapzend:source:snapshot:recursive=true
org.znapzend:source:send:raw=true
org.znapzend:source:send:compressed=true
org.znapzend:source:send:mbuffer:path=/usr/bin/buffer
org.znapzend:source:send:mbuffer:size=1G
org.znapzend:source:send:znap_cmd:pre='echo hello'
org.znapzend:source:send:znap_cmd:post='echo goodbye'
org.znapzend:destination:0:dataset=backup_pool/ROOT/ubuntu
org.znapzend:destination:0:snapshot:plan=1weeks=>1days,1months=>1weeks,1years=>1months
org.znapzend:destination:0:receive:force=true
org.znapzend:destination:0:receive:unmounted=true
org.znapzend:destination:1:dataset=root@bserv:backup_pool/ROOT/ubuntu
org.znapzend:destination:1:snapshot:plan=1months=>1weeks,1years=>1months
org.znapzend:destination:1:receive:force=true
org.znapzend:destination:1:receive:unmounted=true

Still reasonably readable and still objecty like in organization.

Tobias Oetiker
@oetiker
nice
and then the values 256 should make for quite some depth
pspikings
@pspikings
yes, that looks better.... the options are really all destination related though (you might want to send raw to one destination and not another) so the send options could be specified for each destination

so what about:
org.znapzend:dst:<name>
org.znapzend:dst:<name>:sendRaw
org.znapzend:dst:<name>:plan
etc....

the current org.znapzend:dst_<name> could be read as org.znapzend:dst:<name> for backwards compatibility

(and org.znapzend:dst_<name>_plan -> org.znapzend:dst:<name>:plan too)
Tobias Oetiker
@oetiker
yes going all : makes sense
minorsatellite
@minorsatellite
When running znapzend in debug mode, there is some output that reads [debug] cleaning up snapshots on poolname/dataset. Which datasets are being deleted and how are they being selected by znapzend?
Tobias Oetiker
@oetiker
according to your retention rules for the source
bluesbro1982
@bluesbro1982
I'm sorry to possibly hijack the chat with a technical question, but is there a way to have znapzend initiate the replication portion of the festivities from the remote server? Similar to what you can do with zrep? I think I'd like to help with some documentation and use cases if this is possible.
My use case is that I'd like my backup vault to have privileges to pull but not have any other server with write access to push in.
Tobias Oetiker
@oetiker
if you can describe your scenario in more technical terms, I might be able to suggest ...
ahhhh :)
bluesbro1982
@bluesbro1982
I have all my zvols encrypted, and would be doing snapshots / backups raw, such that the backup machine could pull without ever having the keys
Tobias Oetiker
@oetiker
at the moment znapzend can not be reversed ... the idea is that the live copy is the most valuable and may thus not be accessed from the outside ... there has been talk about possibilities to reverse the logic, but no one with coding abilities and design abilities has kept at it ...
bluesbro1982
@bluesbro1982
gotcha. yeah, i just found the open request on github
Tobias Oetiker
@oetiker
znapzend master has support for backup of encrypted volumes
(so that they stay encrypted at the remote end)
bluesbro1982
@bluesbro1982
I saw that - the raw flag is exactly what I needed
the code is mostly python, right? the last scripting stuff I did was perl waaaay back in the day
Tobias Oetiker
@oetiker
the code is all perl
bluesbro1982
@bluesbro1982
What would budget be to contribute to the project and help implement this?
hmm
I just don't have the time right now, but I'm in a place in my life where I'm no longer surviving on ramen...
Tobias Oetiker
@oetiker
the first step would be to open an issue (or maybe there is already one) and propose a design on how this would work on a logical and configuration level
yes the time issue ... that seems to be everybodies problem these days ... mine too
bluesbro1982
@bluesbro1982
oetiker/znapzend#158
Alright, let me check out the code and see if I can conceptualize how this would work.
for reference this is how zrep does it, but the backend architecture is different I'm sure
http://www.bolthole.com/solaris/zrep/zrep.documentation.html#backupserver
Rivqua
@Rivqua
Hi :)
I have a question I can't figure out the answer to, I've setup znapzend, and it works. I am wondering though, how do I configure the features? like --features=skipIntermediates,compressed ?
Jonathan Duggan
@jduggan69_gitlab
Hey guys - is there any literature anywhere on integrating or somehow monitoring a znapzend environment ? i have an environment of around 30 sans currently using homebrew scripts & sending info to a central api for a webfrontend to quickly overlook. its turned into a huge chore so keen to move to something like zsnapzend but wondering if such a thing exists already for the monitoring side ?
Tobias Oetiker
@oetiker
there is no such document, but basically you can parse syslog output
felixn-dk
@felixn-dk
Hi I get this error when I try to modify and list : zLog must be specified at creation time
stellarpower
@stellarpower
Having the same. Is this because it can't connect to syslogd?
Carsten Härle
@chaerle

What does the znapzend 20.0 error message mean?

cannot send 'xxxxxx': I/O error
ERROR: cannot send snapshots to xxxxx on yyyy
send/receive for data1 failed: Can't call method "message" without a package or object reference at /usr/lib/x86_64-linux-gnu/ZnapZend.<

storkone
@storkone
Hi, in lib/ZnapZend.pm $refreshBackupPlans is called during init. That sub creates (to be replicated) zfs datasets at the destination if they don't exist. However the pre-send-command and post-send-command's (if set and not off) are also executed. I find this a bit misleading because the documentation describes a valid example (bringing a zpool online) but other use cases come to mind, like running some job against the transferred snapshot. But that like the parameter name implies, run that job only before and after zfs send do not run it after "refresh" Yes the environment variable WORKER is appended with "-refresh" to distiguish both cases but that's nowhere documented.
Please let me know whether you want the documentation updated or split the pre/post-send-command in the "refresh" and send versions. Which is a bit cleaner than checking some environment variable for a string.
Tobias Oetiker
@oetiker
if you have an idea on how to make this cleaner without breaking existing implementations ...
otherwise improvements for PR the documentation will be fine
storkone
@storkone
Now a design question. Why does the pre/post-snap-cmd includes the snapshot name in an environment variable for the scripts and the pre/post-send-cmd doesn't?
Now the script doesn't know which snapshot was send. I can think of various (edge) cases where something might have changed between the actual send and executing the script. I've hacked this variable in the various sub's to get it from $createSnapshot via the workers to $sendRecvCleanup. If interested I could create a pull request.
I need post processing of the send snapshot, not a snapshot. Another snapshot might have been created, this one might have been deleted. So just processing the last snapshot won't do it. Processing get's done in [dataset]/.zfs/snapshot/[snapshot name]
@oetiker , unfortunately I don't. But I was hoping that you might see a solution.
Tobias Oetiker
@oetiker
I see no problem in providing additional information via environment variables, so please proved a PR!
Kevin Liu
@pot8to
Hey, is there any support in znapzend for deleting local snapshots while keeping remote ones? I'm worried about disks filling up with old snapshots.
Tobias Oetiker
@oetiker
that is the whole point of znapzend ... you configure exactly howmany snapshots you want to keep ... local and remote
Manuel Oetiker
@moetiker
Every location has his own plan ....
src = fast/home-a
src_plan = 1day=>30minutes,7days=>2hours,30days=>4hours
Kevin Liu
@pot8to
yup, thanks!
Carsten Härle
@chaerle
I try to send an unencrypted hierarchy to an encrypted one. The znapzend daemon fails with an error, that it cannot destroy the transfer root encrypted File system with "zfs receive -F" and terminates. However with --runonce znapzend continues with the sub datasets and transfers it correctly. It there a means to have the daemon work also?