Partitioning Special vDEV on Boot Pool - Not Utilizing SVDEV
I have partitioned off ~30G for the Boot pool & 200G for the Special VDEV + Small Blocks on my 3-way mirror but small files and metadata are not being fully written to the Special VDEV.
My expectation is that all blocks <32K should be put in the Special VDEV as configured below:
```sh $ zfs get special_small_blocks tank NAME PROPERTY VALUE SOURCE tank special_small_blocks 32K local ```
```sh # NOTE: rpool mirror-0 are the same drives as special mirror-2, # only that they are different partitions
# zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 28.5G 14.1G 14.4G - - 60% 49% 1.00x ONLINE - mirror-0 28.5G 14.1G 14.4G - - 60% 49.5% - ONLINE ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508033-part3 29.0G - - - - - - - ONLINE ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508401-part3 29.0G - - - - - - - ONLINE ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508422-part3 29.0G - - - - - - - ONLINE tank 25.6T 10.1T 15.5T - - 9% 39% 1.00x ONLINE - mirror-0 10.9T 4.21T 6.70T - - 23% 38.6% - ONLINE wwn-0x5000cca253c8e637-part1 10.9T - - - - - - - ONLINE wwn-0x5000cca253c744ae-part1 10.9T - - - - - - - ONLINE mirror-1 14.5T 5.88T 8.66T - - 0% 40.4% - ONLINE ata-WDC_WUH721816ALE6L4_2CGRLEZP 14.6T - - - - - - - ONLINE ata-WUH721816ALE6L4_2BJMBDBN 14.6T - - - - - - - ONLINE special - - - - - - - - - mirror-2 199G 12.9G 186G - - 25% 6.49% - ONLINE wwn-0x5002538c402f3ace-part4 200G - - - - - - - ONLINE wwn-0x5002538c402f3afc-part4 200G - - - - - - - ONLINE wwn-0x5002538c402f3823-part4 200G - - - - - - - ONLINE ```
I simulated metadata operations with the following `fio` parameters which creates 40000 4k files and reads through them:
```sh DIR=/tank/public/temp
fio --name=metadata \ --directory=$DIR \ --nrfiles=10000 \ --openfiles=1 \ --file_service_type=random \ --filesize=4k \ --ioengine=sync \ --rw=read \ --bs=4k \ --direct=0 \ --numjobs=4 \ --runtime=60 \ --time_based \ --group_reporting
```
The issue is that for some reason the HDD pool is being taxed while the Special VDEV remains low utilization if at all via `iostat -xys --human 1 1` or `zpool iostat -v 1`. I have fully flushed ARC and recreated the files after `rm -f $DIR` with no success.
My question is, why are my small files not being written to the SVDEV and instead the HDD pool? Fresh Proxmox 9.1 & ZFS 2.3.4