RedditPP Mascot

RedditPP

r/zfs • … • u/csdvrx

With old backups, how to check for silent corruption related to zfs_dmu_offset_next_sync?

A scrub may not be sufficient, but due to crazy paranoid data policies, I fortunately have backups for each of the last 18 months.

Some of these backups are even stored on non-ZFS filesystems, but unfortunately, they were all sourced from ZFS, so they are now all suspect of silent corruption.

However, given that the bug seems probabilistic:

- it only increased in probability after 2.1.4 with [commit 9f69435](https://github.com/openzfs/zfs/commit/9f6943504aec36f897f814fb7ae5987425436b11) and [commit 05b3eb6 in master] (https://github.com/openzfs/zfs/commit/05b3eb6d232009db247882a39d518e7282630753), - it became much more likely with block cloning which fortunately was never deployed (we're still on 2.1.13 as of today)

This means older backups may have fewer corrupted files (or none at all), and could be used for restoring.

The difficulty is finding which files are silently corrupted in a current ZFS filesystem.

[Given the github discussion for #15526](https://github.com/openzfs/zfs/issues/15526), I think looking for chunks filled with 0 is a strategy, but they may normally exist inside files.

Could anyone recommend a tool that could be used as a first pass to:

- store in a SQLite database the filename, path, ctime, atime, size, checksum (md5 or sha256) of files from supposedly good backups - compare that to a mounted zfs filesystem to warn about which files are suspicious

I think I could write something using the list of files containing blocks of zeroes, but:

- only checking for zeroes may be both of low specificity (as the github discussion also mentions chunks can be repeated within the file) and of low sensibility (as some files might contain blocks of zeroes - checking for repeated chunks of unknown sizes and startpoint within the file would be algorithmically complex

So ideally, this would be complemented by other checks (like size, known checksum from old backups etc), in a decision tree designed more specifically for this zfs bug.

If anyone has good ideas for the design, I'm all ears!

— 0