Permanent errors <metadata>:<0x611> after losing cache/log

[Harvie]

Hello,
several months ago i've lost my cache and log device to failed SSD. Everything seems OK since then, but in zpool status i see following metadata errors.

# zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Jun 18 11:50:56 2020
        2.16T scanned at 21.9M/s, 1.66T issued at 16.9M/s, 2.16T total
        2.84T resilvered, 76.81% done, 0 days 08:38:35 to go
config:

        NAME                                 STATE     READ WRITE CKSUM
        tank                                 ONLINE       0     0     0
          mirror-0                           ONLINE       0     0     0
            wwn-0x0000000000000000           ONLINE       0     0     3  (resilvering)
            wwn-0x0000000000000000           ONLINE       0     0    48  (resilvering)
            wwn-0x0000000000000000           ONLINE       0     0     0  (resilvering)
        logs
          ata-0000000000000000-0000000000000000-part1  ONLINE       0     0     0
        cache
          ata-0000000000000000-0000000000000000-part2  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0xe>
        <metadata>:<0x611>
        <metadata>:<0x29a>
        <metadata>:<0x1aa>

Scrubbing did not helped. It is usualy reccomended to destroy the pool, create it again and restore from backup. Which is something i would really prefer not to do. Certainly not on zettabyte of data. Since the corruption seems to be relatively minor, is there something i can do to restore the pool consistency without having to recreate whole pool... Eg. overwrite some files with corrupted metadata or recreate just one specific dataset, which is to blame. Is there way to tell, what that <metadata>:<0x611> hex code belongs to, so i can overwrite it?

I tried to do find /tank -inum 1553 (where 1553 is decimal notation of 0x611 hex) and it found several files with this inode number in the pool. (i guess the inode number can be reused several times as long as it is used only once in each dataset) however i am not sure if this is helpful or not...

BTW if i do send/receive of such pool, will the metadata corruption replicate as well?

[richardelling]

@Harvie the checksum errors under the mirror are not related to log or cache.

Since this is in metadata, zdb is a better tool:
zdb -dd tank 0x611
should provide more details. More -d options goes further into detail

How corrupted data is handled by send is configurable. However, it likely doesn't apply to this case because much metadata is not sent. For reference,
https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/ZFS%20on%20Linux%20Module%20Parameters.html#zfs-send-corrupt-data

Lastly,

• "permanent errors" are not always permanent. It depends on the storage subsystem.
• at no time did the zpool status command recommend destroying the pool. Such a message can occur due to other issues, but typically not for isolated corruption that this pool is demonstrating.