How can 2 new identical pools have different free space right after a zfs send|receive giving them the same data?
Hello
For the 2 new drives having the exact same partitions and number of blocks dedicated to ZFS, I have very different free space, and I don't understand why.
Right after doing both `zpool create` and `zfs send | zfs receive`, there is the exact same 1.2T of data, however there's 723G of free space in the drive that got its data from rsync, while there is only 475G in the drive that got its data from `zfs send | zfs receive` of the internal drive:
$ zfs list NAME USED AVAIL REFER MOUNTPOINT internal512 1.19T 723G 96K none internal512/enc 1.19T 723G 192K none internal512/enc/linx 1.19T 723G 1.18T /sysroot internal512/enc/linx/varlog 856K 723G 332K /sysroot/var/log extbkup512 1.19T 475G 96K /bku/extbkup512 extbkup512/enc 1.19T 475G 168K /bku/extbkup512/enc extbkup512/enc/linx 1.19T 475G 1.19T /bku/extbkup512/enc/linx extbkup512/enc/linx/var/log 284K 475G 284K /bku/extbkup512/enc/linx/var/log
Yes, the varlog dataset differs by about 600K because I'm investigating this issue.
What worries me is the 300G difference in "free space": that will be a problem, because the internal drive will get another dataset that's about 500G.
Once this dataset is present in internal512, backups may no longer fit in the extbkup512, while these are identical drives (512e), with the exact same partition size and order!
I double checked: the ZFS partition start and stop at exactly the same block: start=251662336, stop=4000797326 (checked with gdisk and lsblk) so 3749134990 blocks: 3749134990 *512/(1024^3) giving 1.7 TiB
At first I thought about difference in compression, but it's the same:
$ zfs list -Ho name,compressratio internal512 1.26x internal512/enc 1.27x internal512/enc/linx 1.27x internal512/enc/linx/varlog 1.33x extbkup512 1.26x extbkup512/enc 1.26x extbkup512/enc/linx 1.26x extbkup512/enc/linux/varlog 1.40x
Then I retraced all my steps from the zpool history and bash_history, but I can't find anything that could have caused such a difference:
- Step 1 was creating a new pool and datasets on a new drive (internal512)
zpool create internal512 -f -o ashift=12 -o autoexpand=on -o autotrim=on -O mountpoint=none -O canmount=off -O compression=zstd -O xattr=sa -O relatime=on -O normalization=formD -O dnodesize=auto /dev/disk/by-id/nvme....
zfs create internal512/enc -o mountpoint=none -o canmount=off -o encryption=aes-256-gcm -o keyformat=passphrase -o keylocation=prompt
zfs create -o mountpoint=/ internal512/enc/linx -o dedup=on -o recordsize=256K
zfs create -o mountpoint=/var/log internal512/enc/linx/varlog -o setuid=off -o acltype=posixacl -o recordsize=16K -o dedup=off
- Step 2 was populating the new pool with an rsync of the data from a backup pool (backup4kn)
cd /zfs/linx && rsync -HhPpAaXxWvtU --open-noatime /backup ./ (then some mv and basic fixes to make the new pool bootable)
- Step 3 was creating a new backup pool on a new backup drive (extbkup512) using *the EXACT SAME ZPOOL PARAMETERS*
zpool create extbkup512 -f -o ashift=12 -o autoexpand=on -o autotrim=on -O mountpoint=none -O canmount=off -O compression=zstd -O xattr=sa -O relatime=on -O normalization=formD -O dnodesize=auto /dev/disk/by-id/ata...
- Step 4 was doing a scrub, then a snapshot to populate the new backup pool with a `zfs send|zfs receive`
zpool scrub -w internal512@2_scrubbed && zfs snapshot -r internal512@2_scrubbed && zfs send -R -L -P -b -w -v internal512/enc@2_scrubbed | zfs receive -F -d -u -v -s extbkup512
And that's where I'm at right now!
I would like to know what's wrong. My best guess is a silent trim problem causing issues to zfs: doing `zpool trim extbkup512` fail with 'cannot trim: no devices in pool support trim operations', while nothing was reported during the zpool create
For alignment and data recue reasons, ZFS does not get the full disks (we have a mix, mostly 512e drives and a few 4kn): instead, partitions are created on 64k alignment, with at least one EFI partition on each disk, then 100G to install whatever if the drive needs to be bootable, or to do tests (this is how I can confirm trimming works)
I know it's popular to give entire drives to ZFS, but drives sometimes differs in their block count which can be a problem when restoring from a binary image, or when having to "transplant" a drive into a new computer to get it going with existing datasets.
Here, I have tried to create a non zfs filesystem on the spare partition to do a `fstrim -v` but it didn't work either: fstrim says 'the discard operation is not supported', while it works on Windows with 'defrag and optimize' for another partition of this drive, and also manually on this drive if I trim by sector range with `hdparm --please-destroy-my-drive --trim-sector-ranges $STARTSECTOR:65535 /dev/sda`
Before I give the extra 100G partition to ZFS, I would like to know what's happening, and if the trim problem may cause free space issues later on during a normal use.