I have successfully setup Debian stretch on ZFS, including the root file system. Things are working like expected, and I was thinking that I had understood the basic concepts - until I re-read Sun's ZFS documentation.
My scenario is:
I'd like to prevent (more precisely: detect) silent bit rot
For the moment, I have set up a root pool with one vdev which is a mirror of two identical disks
Of course, I did turn on (i.e. did not turn off) checksums
Now I have come across this document. At the end of the page, they show the output of the
zpool status command for their example configuration,
[...] NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 OFFLINE 0 0 0 48K resilvered [...]
followed by the statement:
The READ and WRITE columns provide a count of I/O errors that occurred on the device, while the CKSUM column provides a count of uncorrectable checksum errors that occurred on the device.
First, what does "device" mean in this context? Are they talking about a physical device, the vdev or even something else? My assumption is that they are talking about every "device" in the hierarchy. The vdev error counter then probably is the sum of the error counters of its physical devices, and the pool error counter probably is the sum of the error counters of its vdevs. Is this correct?
Second, what do they mean by uncorrectable checksum errors? This is a term which I thought is usually used when talking about physical disks, either relating to data transfer from the platter into the disk's electronics, to checksums of physical sectors on the disk or to data transfer from the disk's port (SATA, SAS, ...) to the mainboard (or controller).
But what I am really interested in is whether there have been checksum errors at ZFS level (and not hardware level). I am currently convinced that CKSUM is showing the latter (otherwise, it wouldn't make much sense), but I'd like to know for sure.
Third, assuming the checksum errors they are talking about are indeed the checksum errors at the ZFS level (and not hardware level), why do they only show the count of uncorrectable errors? This does not make any sense. We would like to see every checksum error, whether correctable or not, wouldn't we? After all, a checksum error means that there has been some sort of data corruption on the disk which has not been detected by hardware, so we probably want to change that disk as soon as there is any error (even if the mirror disk can still act as "backup"). So I possibly did not understand yet what exactly they mean by "uncorrectable errors".
Then I have come across this document which is even harder to understand. Near the end of the page, it states
[...] ZFS maintains a persistent log of all data errors associated with a pool. [...]
and then states
Data corruption errors are always fatal. Their presence indicates that at least one application experienced an I/O error due to corrupt data within the pool. Device errors within a redundant pool do not result in data corruption and are not recorded as part of this log. [...]
I am heavily worried about the third sentence. According to that paragraph, there could be two sorts of errors: Data corruption errors and device errors. A mirror configuration of two disks is undoubtedly redundant, so (according to that paragraph) it is no data corruption error if ZFS encounters a checksum error on one of the disks (at the ZFS checksum level, not the hardware level). That means (once more according to that paragraph) that this error will not be recorded as part of the persistent error log.
This would not make any sense, so I must have got something wrong. For me, the main reason for switching to ZFS was its ability to detect silent bit rot on its own, i.e. to detect and report errors on devices even if those errors did not lead to I/O failures at the hardware / driver level. But not including such errors in the persistent log would mean losing them upon reboot, and that would be fatal (IMHO).
So eventually Sun has chosen worrying wording here, or I have misunderstood some concepts (not being a native English speaker).