Samba or pool performance slow 91% full pool?

Hello I used to run geli encryption but somewhat recently switched to the native zfs one. About the same time my samba performance has been really slow, I'll have 2 second delay to list contents of a folder over SMB.

I just ran a FIO test and I guess maybe its not the pool thats bad but samba? Also the disks are quite full now 91% used.

`fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=rw --size=50g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=16 --runtime=120 --group_reporting --output=/root/test.txt` `TEST: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=16` `...` `fio-3.28` `Starting 16 processes` `TEST: Laying out IO file (1 file / 51200MiB)`

`TEST: (groupid=0, jobs=16): err= 0: pid=39360: Sun Aug 18 20:47:47 2024` ` read: IOPS=17.7k, BW=2207MiB/s (2314MB/s)(259GiB/120001msec)` ` clat (usec): min=5, max=1258.9k, avg=89.14, stdev=2882.99` ` lat (usec): min=5, max=1258.9k, avg=89.31, stdev=2882.99` ` clat percentiles (usec):` ` | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17],` ` | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 28], 60.00th=[ 33],` ` | 70.00th=[ 38], 80.00th=[ 45], 90.00th=[ 63], 95.00th=[ 221],` ` | 99.00th=[ 840], 99.50th=[ 1254], 99.90th=[ 4686], 99.95th=[10552],` ` | 99.99th=[57934]` ` bw ( MiB/s): min= 565, max= 9535, per=100.00%, avg=2236.88, stdev=90.32, samples=3776` ` iops : min= 4519, max=76272, avg=17888.70, stdev=722.56, samples=3776` ` write: IOPS=17.7k, BW=2211MiB/s (2318MB/s)(259GiB/120001msec); 0 zone resets` ` clat (usec): min=8, max=1261.8k, avg=808.71, stdev=3420.34` ` lat (usec): min=9, max=1261.8k, avg=811.72, stdev=3420.41` ` clat percentiles (usec):` ` | 1.00th=[ 14], 5.00th=[ 36], 10.00th=[ 59], 20.00th=[ 330],` ` | 30.00th=[ 441], 40.00th=[ 570], 50.00th=[ 685], 60.00th=[ 799],` ` | 70.00th=[ 930], 80.00th=[ 1156], 90.00th=[ 1565], 95.00th=[ 1926],` ` | 99.00th=[ 2671], 99.50th=[ 2933], 99.90th=[ 5473], 99.95th=[12256],` ` | 99.99th=[59507]` ` bw ( MiB/s): min= 621, max= 9565, per=100.00%, avg=2240.59, stdev=90.24, samples=3776` ` iops : min= 4968, max=76517, avg=17918.31, stdev=721.88, samples=3776` ` lat (usec) : 10=0.19%, 20=15.90%, 50=30.79%, 100=5.88%, 250=2.58%` ` lat (usec) : 500=10.99%, 750=11.11%, 1000=9.07%` ` lat (msec) : 2=11.25%, 4=2.11%, 10=0.07%, 20=0.02%, 50=0.02%` ` lat (msec) : 100=0.01%, 250=0.01%, 1000=0.01%, 2000=0.01%` ` cpu : usr=0.98%, sys=7.69%, ctx=2233590, majf=0, minf=0` ` IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%` ` submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%` ` complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%` ` issued rwts: total=2118698,2122109,0,0 short=0,0,0,0 dropped=0,0,0,0` ` latency : target=0, window=0, percentile=100.00%, depth=16`

`Run status group 0 (all jobs):` ` READ: bw=2207MiB/s (2314MB/s), 2207MiB/s-2207MiB/s (2314MB/s-2314MB/s), io=259GiB (278GB), run=120001-120001msec` ` WRITE: bw=2211MiB/s (2318MB/s), 2211MiB/s-2211MiB/s (2318MB/s-2318MB/s), io=259GiB (278GB), run=120001-120001msec`