RAIDZ Expansion: To do or not to do
I think I have a borderline situation as to whether I should attempt a raidz expansion or not.
I have a 6x14TB Z2 vdev pool, which I thought I was going to be good with for a while, but then I grabbed a pair of 28TB disks for just over $300 each last year. And now I want more... I want to expand it from 56TB usable to 84TB by leveraging the two new spindles; I will partition 14TB out of each new 28TB disk to build a 8-wide Z2.
I have 19 or so TB (say 20) utilized in the 6 wide pool.
My new disks will give me 2x14TB partitions of scratch. I have available to vacate from my older disks, 8, 6, 2, 2, 2 TB. I can partition the 8 into a 6 and 2, giving me the ability to make a scratch pool that has 1-disk redundancy, e.g. with a topology made of mirrors of 6TB and 14TB and the 4 remaining slices of 2TB into something... since i only need 20, I may just do mirrors instead of raidz with those 2TB disks/parts.
So then the idea would be to spin that scratch pool up, send/recv from my 6-wide pool into it, and i will have redundancy present in all copies of my data at this point which will fit into the 24 or so TB of scratch pool space i'm creating this way.
Then I can destroy the 6 wide and create the 8 wide (by adding in the two new 14TB partitions) and send/recv a final time into the 8-wide. I estimate my 20TB should take 2 days to transfer so this will take like a whole week. I base this estimate on the fact that I just used rsync to complete my long running 14tb mirror to 6 wide 14tb z2 expansion I was doing (i kept one of the original pool's mirror disks around for validation) and the 14 or so TB took 37 hours to scan through with checksums. OTOH i have its resilver proceeding and the estimate puts it at about 8 hours total for 14TB of content...
The alternative is to just use RAIDZ expansion with my 2 additional partitions. It's a lot cleaner and I could also then delay fully cleaning out my older smaller disks which is a plus. I figure if the raidz expansion is going to be hands off it should lead to better quality of life, even if it takes longer than a week to crunch through the two 14TB partitions I'm looking to add.
What would you do in this situation? I know that expansion will leave me with unevenly distributed data that might be a bit awkward to fully redistribute. The USUAL situation is that the backup is there and it's easy, but in this case it's borderline. It's definitely a frankenstein and I have to design and build the frankenstein before I can use it. That part is fun for me though. It does have full redundancy, just not very high quality redundancy...
I already did the aforementioned shuffle where I was able to take a 2x14TB mirror that was completely full and with 4 fresh 14TB disks was able to finagle it into a final state of 6x14TB Z2 without any need for additional backup, which was until this upcoming one the most complex zfs operation i've attempted. luckily due to good planning it went smoothly, and I expect either of these paths I take will also go smoothly, I guess the question is maybe which is both safer and easier (as it would be silly to go against the option that is both easier and safer) and I suspect that the answer will be the expansion rather than the scratch disk frankensteining. It's just that most threads i see here say that expansion is a pain, so I'd like to learn more about what exactly makes it a pain?
I also have an extra wrinkle with the "traditional" pool upgrade: my 28TB disks' spindles will be shared by the scratch pool and the target 8-wide pool. This will make the final replication really slow and probably noisy even though it should not compromise safety much. I also wonder if it would overly wear the disk write heads with unrelenting thrash.
I guess I may as well go and clean up my spare disks to make the scratch pool so i can leverage it as an additional backup and i might still try the raidz expansion. So... the maximum pain and absolute maximum safety route.
I'm not really interested in cloud or offsite backup yet. this is too much data and will take too long and cost too much. i want to be efficient about putting my dollars into real hardware i can leverage for work, not renting stuff. Long term I do want that but it's for when I get around to reorganizing the stuff I actually care about into its own dataset so i can replicate that (and only that) off to offsite and cloud.