Any way to improve ZFS serial write performance ( 50Mb/s) on a NVMe pool?

On a decent Xeon server (128 Gb ECC, NVMe...) where an important dataset was removed by mistake, I had to restore about 1Tb of data from an emergency offsite backup.

To go faster, I simply removed a NVMe from a mirror pool that contains backup datasets, drove to the server with this NVMe in an ESD bag, physically plugged the NVMe in the server, imported this pool, created a new dataset (1M recsize) on the original server pool where the dataset had been removed, and copied the files with mc.

I used mc just because it has a cute progress bar giving the ETA, but it may have been a bad idea since the reported write performance was terrible: starting at 300 Mb/s, it stabilized at 50 Mb/s. I thought mc was reporting it wrong, but it must have been right since restoring the data took several hours!

Since I had hours to wait, I tried to investigate the issue during the restore: I found a SATA SSD had been wrongly added to the NVMe mirror pool, so I took it offline from this pool but it didn't help.

Then I thought the mirror pool might be delayed by another drive, I tried to leave only 1x NVMe drive in the pool, but it didn't help either.

Is it possible the SATA being wrongly present in the NVMe pool, even when marked offline, was the cause of this bad write performance?

Is there anything I am doing wrong? Can the time to restore from backups be improved?

On the server:

- the partitions are physically aligned to start a 64k, the NVMe is not QLC, it is not HMB either: it has DRAM cache, and the large recordsize (1M) seems appropriate for the files (about 50M each) - for the ZFS NVMe pool, it's just multiple mirrors (no draid or anything complicated), with ashift=12; by default the datasets are encrypted and compressed (zstd) but there's no "costly" option like dedup.

There are a few Optanes left, I could use one for a ZIL, but if sync is the issue, temporarily using nosync when restoring a large backup may be simpler.

The server also has a SATA SSD pool with multiple mirrors, used for backups and tests.

Next week, I will take a few drives off the SATA SSD pool and experiment with different ZFS settings (ex: no compression), and with other filesystems too, in order to have a few reference points:

- XFS over mdadm, - NTFS with the new Paragon kernel driver, - maybe even bcacheFS.

However, if a SATA SSD pool can beat a NVMe pool, there's something very wrong!

Any hint or suggestion would be appreciated, as I would prefer to keep using ZFS on the servers.