Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
minorsatellite
@minorsatellite
No, hold is available in all ZFS code bases (Illumos, FreeBSD, Linux). I will take a look at bookmarks, thanks.
pspikings
@pspikings
Hi..... I want to replicate my encrypted snapshots to a remote pool and have them stored encrypted without the key. I think that just adding -w to the zfs send options is all that's required but znapzend can't set that. Would you be interested in a PR to do that and is there anything to watch out for? Would you think it best just to support -w or have a config setting to set arbitrary extra flags? :)
Tobias Oetiker
@oetiker
sure
to keep consistent, an explicit option might be best .. wondering though if the option could be set automatically
pspikings
@pspikings
good point, it could be set if the source dataset is encrypted but then you might get people wanting the zfs default behavior of decrypting before sending so the backup doesn't require the key to read :)
Tobias Oetiker
@oetiker
I would rather make this behavior configurable :)
but that is probably counter intuitive
pspikings
@pspikings
PR opened.... it's working as expected here :) See PR comment about future improvement
Also.... belated thanks for RRDtool, I used that lots in a previous job, very useful! :)
Tobias Oetiker
@oetiker
glad you like it
James Crocker
@james-crocker
Hello - am enjoying znapzend! I've encrypted datasets and was messing with the code base to support sending raw streams only to do a fresh pull and see the sendRaw feature - yeah! So, while I don't have to continue pursing enabling that feature myself - I'm still looking at how best to manage the ZFS pools for send/recv. Specifically, I really like that the 'config' is stored in the zfs properties - keeping everything nicely self-contained. However, 'features' are not stored along with the other org.znapzend ZFS properties. Currently then, to enable the sendRaw feature it must be passed to the $ZNAPZENDOPTIONS for the znapzend.service unit's ExecStart (and/or configured in an EnvironmentFile). Consequently if I want one particular pool to sendRaw, while another doesn't, it would require creating multiple service unit files with or without the --featrues options sent to znapzend. I'd like take on the task of embedding the features as org.znapzend property and reference it when znapzend is called - perhaps allowing the cli --features to override any ZFS org.znapzend:features property. But, I thought before I invest any goodly time and effort to do so - Were there any reasons that the cli --features option is not already stored as an ZFS org.znapzend:features property for the datasets? Thanks!
Tobias Oetiker
@oetiker
yes having features per-destination would be great we just have no found a good sustainable way to have that yet. but maybe adding a per dest feature property with key value pairs might be a nice way of doing it ...
another much more radical aproach we thought about was to have the config as json, base64 encode it and store it in a buch of properties (no sure how much data a single property can take)
note that some features will be per source, some per dest
a pr for that would be great
James Crocker
@james-crocker

Haven't found anything specific from ZoL; but if it follows Oracle ZFS ...

https://docs.oracle.com/cd/E19253-01/819-5461/gazss/index.html
...
User property names must conform to the following conventions:
-They must contain a colon (':') character to distinguish them from native properties.
-They must contain lowercase letters, numbers, or the following punctuation characters: ':', '+','.', '_'.
-The maximum length of a user property name is 256 characters.

The values of user properties must conform to the following conventions:
-They must consist of arbitrary strings that are always inherited and are never validated.
-The maximum length of the user property value is 1024 characters.

So, a json config may bump up to the max length sooner than later.

And regardless, imho, it's nice to be able to review/see the zfs property values in human readable form - without having to jump through hoops to decode a base64 string. And, even if json stored plain-text, it would be challenging to quickly parse visually.

For what it's worth - is this sort of the thought on a json config?

{ "source": { "enabled": true, "dataset": "rpool/ROOT/ubuntu", "snapshot": { "set_property": "property1=value1,property2=value2", "tsformat": "%Y-%m-%dT%H:%M:%SZ", "plan": "1hours=>15minutes,1days=>1hours,1weeks=>1days,1months=>1weeks,1years=>1months", "recursive": true }, "send": { "raw": true, "compressed": true, "mbuffer": { "path": "/usr/bin/mbuffer", "size": "1G" }, "znap_cmd": { "pre": "echo hello", "post": "echo goodbye" } } }, "destinations": [{ "dataset": "backup_pool/ROOT/ubuntu", "snapshot": { "plan": "1weeks=>1days,1months=>1weeks,1years=>1months", "receive": { "force": true, "unmounted": true, "set_property": "property1=value1,property2=value2" } } }, { "datasets": "root@bserv:backup_pool/ROOT/ubuntu", "snapshot": { "plan": "1months=>1weeks,1years=>1months", "receive": { "force": true, "unmounted": true, "set_property": "property1=value1,property2=value2" } } }, { "datasets": "root@bserv2:backup_pool/ROOT/ubuntu", "snapshot": { "plan": "1years=>1months", "receive": { "force": true, "unmounted": true, "set_property": "property1=value1,property2=value2" } } }, { "datasets": "root@bserv3:backup_pool/ROOT/ubuntu", "snapshot": { "plan": "1hours=>15minutes,1days=>1hours", "receive": { "force": true, "unmounted": true, "set_property": "property1=value1,property2=value2" } } } ] }

Tobias Oetiker
@oetiker
yes something along these lines ... but the points you listed are somethinf which prevented us from takeing that route
James Crocker
@james-crocker
And just FYI I did put that json into a zfs property to see what it'd look like... The wc -c character count on that json is 1193 - So, ZoL property values can apparently exceed 1024 char. But, as discussed challenging to read on a zfs get all - and throws off the output a bit too. The ZoL zfs property name is constrained to 256 chars.
James Crocker
@james-crocker

So, the 256 char allowed for zfs property name is quite expansive. What if - a pseudo like json zfs property? Something like

org.znapzend:source:enabled=true
org.znapzend:source:snapshot:plan=1hours=>15minutes,1days=>1hours,1weeks=>1days,1months=>1weeks,1years=>1months
org.znapzend:source:snapshot:tsformat=%Y-%m-%dT%H:%M:%SZ
org.znapzend:source:snapshot:recursive=true
org.znapzend:source:send:raw=true
org.znapzend:source:send:compressed=true
org.znapzend:source:send:mbuffer:path=/usr/bin/buffer
org.znapzend:source:send:mbuffer:size=1G
org.znapzend:source:send:znap_cmd:pre='echo hello'
org.znapzend:source:send:znap_cmd:post='echo goodbye'
org.znapzend:destination:0:dataset=backup_pool/ROOT/ubuntu
org.znapzend:destination:0:snapshot:plan=1weeks=>1days,1months=>1weeks,1years=>1months
org.znapzend:destination:0:receive:force=true
org.znapzend:destination:0:receive:unmounted=true
org.znapzend:destination:1:dataset=root@bserv:backup_pool/ROOT/ubuntu
org.znapzend:destination:1:snapshot:plan=1months=>1weeks,1years=>1months
org.znapzend:destination:1:receive:force=true
org.znapzend:destination:1:receive:unmounted=true

Still reasonably readable and still objecty like in organization.

Tobias Oetiker
@oetiker
nice
and then the values 256 should make for quite some depth
pspikings
@pspikings
yes, that looks better.... the options are really all destination related though (you might want to send raw to one destination and not another) so the send options could be specified for each destination

so what about:
org.znapzend:dst:<name>
org.znapzend:dst:<name>:sendRaw
org.znapzend:dst:<name>:plan
etc....

the current org.znapzend:dst_<name> could be read as org.znapzend:dst:<name> for backwards compatibility

(and org.znapzend:dst_<name>_plan -> org.znapzend:dst:<name>:plan too)
Tobias Oetiker
@oetiker
yes going all : makes sense
minorsatellite
@minorsatellite
When running znapzend in debug mode, there is some output that reads [debug] cleaning up snapshots on poolname/dataset. Which datasets are being deleted and how are they being selected by znapzend?
Tobias Oetiker
@oetiker
according to your retention rules for the source
bluesbro1982
@bluesbro1982
I'm sorry to possibly hijack the chat with a technical question, but is there a way to have znapzend initiate the replication portion of the festivities from the remote server? Similar to what you can do with zrep? I think I'd like to help with some documentation and use cases if this is possible.
My use case is that I'd like my backup vault to have privileges to pull but not have any other server with write access to push in.
Tobias Oetiker
@oetiker
if you can describe your scenario in more technical terms, I might be able to suggest ...
ahhhh :)
bluesbro1982
@bluesbro1982
I have all my zvols encrypted, and would be doing snapshots / backups raw, such that the backup machine could pull without ever having the keys
Tobias Oetiker
@oetiker
at the moment znapzend can not be reversed ... the idea is that the live copy is the most valuable and may thus not be accessed from the outside ... there has been talk about possibilities to reverse the logic, but no one with coding abilities and design abilities has kept at it ...
bluesbro1982
@bluesbro1982
gotcha. yeah, i just found the open request on github
Tobias Oetiker
@oetiker
znapzend master has support for backup of encrypted volumes
(so that they stay encrypted at the remote end)
bluesbro1982
@bluesbro1982
I saw that - the raw flag is exactly what I needed
the code is mostly python, right? the last scripting stuff I did was perl waaaay back in the day
Tobias Oetiker
@oetiker
the code is all perl
bluesbro1982
@bluesbro1982
What would budget be to contribute to the project and help implement this?
hmm
I just don't have the time right now, but I'm in a place in my life where I'm no longer surviving on ramen...
Tobias Oetiker
@oetiker
the first step would be to open an issue (or maybe there is already one) and propose a design on how this would work on a logical and configuration level
yes the time issue ... that seems to be everybodies problem these days ... mine too
bluesbro1982
@bluesbro1982
Alright, let me check out the code and see if I can conceptualize how this would work.
for reference this is how zrep does it, but the backend architecture is different I'm sure
Rivqua
@Rivqua
Hi :)
I have a question I can't figure out the answer to, I've setup znapzend, and it works. I am wondering though, how do I configure the features? like --features=skipIntermediates,compressed ?