ZFS NVMe Performance on Sync Writes
I'm running a KVM virtual machine on a ZFS on Linux dataset, and I'm having a strange performance issue. With random I/O, if I have sync=standard, it tanks the performance to nearly 1/20th of what I get with sync=disabled. I'm using fio to benchmark the storage and while I expect to lose some performance, this seems like way too much.
The pool configuration:
- 2 x 1 TB Samsung 980 Pro NVMe SSDs, striped (no mirror), no other VDEVs
- Compression on, lz4
- primarycache=metadata
- atime=off
- xattr=sa
- recordsize=16k
The VM is file-based. Currently qcow2 but I tried a raw instance as well and it was more or less the same.
Here is the comparison between sync=standard and sync=disabled while watching iostat during an fio run. Why could the discrepancy be so huge here? All of this testing is inside the VM.
FIO results:
Sync standard:
read: IOPS=1631, BW=6527KiB/s (6683kB/s)(382MiB/60011msec)
write: IOPS=1632, BW=6529KiB/s (6685kB/s)(383MiB/60011msec); 0 zone resets
Sync disabled:
read: IOPS=27.1k, BW=106MiB/s (111MB/s)(6356MiB/60001msec)
write: IOPS=27.1k, BW=106MiB/s (111MB/s)(6350MiB/60001msec); 0 zone resets
sync standard
capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim pool alloc free read write read write read write read write read write read write wait wait
FLASH 1.39T 436G 3.29K 210 163M 10.7M 185us 2ms 170us 148us 2us 951us 391us 2ms 1ms -
FLASH 1.39T 436G 5.19K 284 80.2M 9.32M 198us 41us 180us 38us 15us 2us - - - -
FLASH 1.39T 436G 4.96K 284 76.6M 8.87M 200us 39us 185us 36us 12us 2us - - - -
FLASH 1.39T 436G 5.00K 279 77.2M 9.11M 202us 43us 184us 39us 14us 3us - - - -
FLASH 1.39T 436G 4.86K 273 75.1M 8.91M 197us 42us 182us 39us 12us 2us - - - -
FLASH 1.39T 436G 4.00K 2.47K 61.9M 156M 221us 5ms 201us 495us 14us 214us - 5ms - -
FLASH 1.39T 436G 5K 269 77.3M 8.93M 188us 43us 173us 39us 10us 2us - - - -
FLASH 1.39T 436G 4.87K 271 75.3M 8.87M 199us 43us 185us 40us 11us 2us - - - -
FLASH 1.39T 436G 4.83K 272 74.6M 8.77M 185us 40us 174us 37us 7us 2us - - - -
FLASH 1.39T 436G 5.09K 287 78.7M 9.39M 197us 39us 181us 37us 12us 2us - - - -
FLASH 1.39T 436G 4.56K 2.77K 70.4M 162M 224us 2ms 209us 306us 11us 4us - 2ms - -
FLASH 1.39T 436G 5.00K 281 77.3M 9.06M 200us 42us 183us 38us 14us 2us - - - -
FLASH 1.39T 436G 5.11K 271 79.0M 9.31M 188us 40us 175us 37us 10us 2us - - - -
FLASH 1.39T 436G 4.92K 275 76.0M 8.85M 183us 42us 172us 39us 6us 2us - - - -
FLASH 1.39T 436G 4.85K 277 75.0M 8.88M 180us 39us 169us 37us 7us 2us - - - -
FLASH 1.39T 436G 4.78K 1.31K 73.9M 69.5M 193us 871us 179us 148us 8us 2us - 884us - -
FLASH 1.39T 436G 4.64K 1.44K 71.7M 99.6M 212us 8ms 194us 432us 15us 2us - 9ms - -
FLASH 1.39T 436G 4.97K 279 76.8M 9.13M 178us 42us 168us 38us 6us 2us - - - -
FLASH 1.39T 436G 5.10K 278 78.9M 9.30M 189us 41us 176us 38us 8us 2us - - - -
FLASH 1.39T 436G 5.12K 274 79.3M 9.38M 188us 43us 175us 39us 11us 2us - - - -
FLASH 1.39T 436G 5.00K 275 77.3M 9.17M 191us 40us 175us 38us 13us 2us - - - -
FLASH 1.39T 436G 4.45K 2.33K 68.5M 164M 221us 7ms 204us 417us 15us 2us - 7ms - -
FLASH 1.39T 436G 5.20K 274 80.4M 9.31M 180us 40us 169us 37us 7us 2us - - - -
FLASH 1.39T 436G 5.10K 275 78.9M 9.31M 192us 40us 176us 38us 12us 2us - - - -
FLASH 1.39T 436G 4.91K 276 76.0M 9.03M 184us 39us 172us 37us 7us 2us - - - -
FLASH 1.39T 436G 4.86K 278 75.3M 8.90M 182us 39us 169us 36us 8us 2us - - - -
FLASH 1.39T 436G 4.60K 2.53K 71.1M 164M 213us 6ms 196us 329us 14us 2us - 7ms - -
FLASH 1.39T 436G 5.19K 275 80.4M 9.24M 185us 41us 172us 38us 8us 2us - - - -
FLASH 1.39T 436G 5.17K 272 80.0M 9.27M 191us 44us 179us 41us 10us 2us - - - -
FLASH 1.39T 436G 4.89K 262 75.7M 8.93M 180us 42us 169us 40us 7us 2us - - - -
FLASH 1.39T 436G 5.04K 273 78.0M 9.30M 188us 40us 175us 38us 9us 2us - - - -