How to split volume to subvolumes

[DaLiV]

initialy system has been planned on single zpool/zfs
now comes decision to split datas on some criteria but some TB are used and drive usage is at 75% ...

pool1/ALL-VMs
want to have
pool1/VM0-DATA + pool1/VM1-DATA + pool1/VM2-DATA + pool1/VM3-DATA + pool1/VM4-DATA+...

so every Vm-Data must be backed (sent-received) onto diffirent "cold" server and syncronized with current master on daily basis (there is no plans to shut off current mater)

for copying full - that takes huge amount of time also as sending full to every point even if that is not related to destination over net.

local zfs send-receive will lasts also too long and result requires 2 times of space (and with current usage of pool at ca 75% that is actually even not possible to be done)

so how can to act for splitting such pool into subpools ?

[IvanVolosyuk]

This could be handled by BRT (block cloning). You can just copy from one dataset to the other with 'cp --reflink=auto'. BRT though is not considered production stable at this point and disabled by default on ZFS-2.2, there are a lot of bug fixes on 2.2.3-staging for this feature. The good part is that once you delete the original dataset the reflink count will go to 1 and BRT will be disengaged. The problem is that the feature will be enabled in the pool, so some other code can use reflink copy which can potentially explode the zfs send stream. This can be solved by disabling BRT again via kernel parameters, so no new reflinks can be created. The other problem is that you cannot go back to the older version of ZFS, thus you are stuck with ZFS 2.2 and above, you may want to carefully consider that before doing 'zpool upgrade'. -- Regards, Ivan

…

On Fri, Feb 16, 2024 at 5:00 AM DaLiV ***@***.***> wrote: initialy system has been planned on single zpool/zfs now comes decision to split datas on some criteria but some TB are used and drive usage is at 75% ... pool1/ALL-VMs want to have pool1/VM0-DATA + pool1/VM1-DATA + pool1/VM2-DATA + pool1/VM3-DATA + pool1/VM4-DATA+... so every Vm-Data must be backed (sent-received) onto diffirent "cold" server and syncronized with current master on daily basis (there is no plans to shut off current mater) for copying full - that takes huge amount of time also as sending full to every point even if that is not related to destination over net. local zfs send-receive will lasts also too long and result requires 2 times of space (and with current usage of pool at ca 75% that is actually even not possible to be done) so how can to act for splitting such pool into subpools ? — Reply to this email directly, view it on GitHub <#15900>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HISO2B7TYMEHKSY7V3YTZEKNAVCNFSM6AAAAABDKWVLUSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWGIZDGMRRGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rincebrain]

That's incorrect, yes. --reflink=always will fail on Linux cross-dataset, because the VFS errors on the ioctl before it ever gets to ZFS, but copy_file_range will happily go brrr.

I believe it will always clone unless the blocks are being modified in the same txg, and there's a tunable around that. (The recent changes around block sizes are to stop you cloning things where the recordsizes aren't the same, IIRC.)

[IvanVolosyuk]

Well yes, I believe that code will not clone files smaller than recordsize as the blocksize is not fixed at this point. I reflinked /usr/lib from gentoo unchanged installation, which is 3.4 GB and only 709M of data was cloned, the rest was copied.
Update: I tried to reflink a non-ZFS data. That's explains the confusion. All ZFS data was reflinked successfully cross-dataset.

[rincebrain]

That could be true, but I don't recall anything like that in the code already, since if you modify the file it's already going to write a larger record, the original being in the BRT doesn't really matter.

[rincebrain]

You might want to try the new tunable for whether it falls back to copying or blocks if it might not clone.

[IvanVolosyuk]

That's is how FUD is created. :) The reflinking cross-dataset worked flawlessly when it was actually the same ZFS pool with small and large files reflinked. Obviously, I cannot clone from other filesystems, that's the mistake I made here.

[lundman]

Wouldn't you snapshot master, then zfs clone it to pool1/VM1-DATA.

[IvanVolosyuk]

I don't think this will work with 4 datasets. All of the clones will share the common ancestor which cannot be deleted and we are back to the square one - how to copy the data away to break the dependency. Thus it will become a ticking bomb as common base will be consuming more and more space after changes are made to the clones.

[DaLiV]

master -> snapshot -> clone -> promote
will not allows to delete intermediate snapshots, also as when you using clone as "new master" - all data to old master will not be transfered by any reverse operation (an other VMs must continue use main master without stopping) so can not normally go in such way.

reflinks as mentioned in one if previous comments are not "production-safe"

[rincebrain]

FWIW, #15139 is the only outstanding BRT bug I know about, and even then, I'm not sure that PR is still necessary after the other PR linked from it, I haven't been looking too closely recently.

So it's possible that 2.2.3 or soon thereafter might suggest you can rely on that feature? I genuinely don't know, I haven't been testing or following it that closely, but I haven't recently seen any new reports of fire and brimstone around it.