0

It is known that there are self-healing file systems like e.g. ZFS, Btrfs, bcachefs and self-healing RAM, like e.g. ECC RAM or corresponding software implementations, which can correct single or multiple erroneous bits.

What are there for self-healing file formats or projects to self-healing file formats for standard programs ?

What does a file format mean? P.e:

  • .txt, .doc, tar.lz4, .mp4
0

2 Answers 2

5
+50

Parchives can be used to store additional repair information alongside any kind of file, and the repair files can then be used to reconstruct missing or damaged data in the originals. Some tools include parchive features in their own file formats; for example, DAR can include repair data when creating backups.

RAR files can also be created with accompanying recovery volumes, which allow certain errors to be repaired. (Note that “File repairing” in Wikipedia’s comparison of file archives can be misleading — it refers to the ability to repair archives, not their contents; archives are commonly repaired by deleting corrupted files inside them.)

0
5

Communications Engineer here. So, someone who does noisy communications and recovery of data from corrupted transmissions professionally, you might say.

Since a file doesn't know or care about the channel¹ it's transferred over:

Error correction needs to be designed for the kind of errors you expect. Thus, aside from a few archival formats, putting error correction into the file is a design mistake. You'd want to put that where there's actual knowledge about the kind and probabilities of errors that can occur.
For example,

  • your Wifi stack adds a lot of error correction (and adjustably much!) that reflects exactly the kind of errors that happen "over the air";
  • your hard drive adds a different amount and kind of error correction to counter readout errors;
  • the line card that drives the fiberoptic cable running through the ocean: yet another kind of error correction.

All these could be transporting your file.

When storing or sending a file, the self-correction belongs to the medium controller, not the file.

  • Same why your browser doesn't add 1/4 of your HTTP traffic's bits in redundancy before you use your WiFi to browse the web: That's not the job of the application, link or routing layer, that's the job of the physical layer. How should your browser know whether your WiFi channel is bad (for every bit you send, you need to send 1 additional bit of error correction redundancy) or excellent (only need to send 1 bit of redundancy for every 6 bits of data)?

Hence, making sure that files are intact is job of whatever stores or transports these files, not of the files themselves. You will thus mostly find long-term archival formats and pretty obsolete file formats (as used to transfer larger files than sensible over usenet; those were the times!), both as listed by Stephen's answer.

(NB: there's exceptions to this rule. In some cases, a file format has a specific use that comes with models for error probabilities and resilience needs – as said, for a long-term archival format, adding redundancy to restore after long-time storage makes sense, but honestly, the difference between a read-only filesystem and an archive file format is that you usually don't write an archive directly to a disk volume, nor a file format into a file, but both is done sometimes. For example, SquashFS is clearly a file system, but I send SquashFS images via email more often these days than I send .tar archives. You mention MP4, and that's a container format that itself is not that resilient to bit errors, but the contained multimedia data might be – due to the design of the codec, single bit flips might not matter too much; that makes sense for e.g. voice codecs to be used over low-latency wireless links. But again, that's an exception, and aside from rather badly designed file formats and the aforementioned long-term storage archive formats which kind of play the role of a "canned" file system, you rarely find error correction information in file formats. You will find a lot of error detection in the shape of short checksums; that's a very cheap and easy way of detecting errors and then shifting the blame and the job to correct down to the transport/storage medium.)


¹ With "channel" we mean anything that can corrupt anything; for example, a serial line that randomly flips a bit with probability of 1/1000 is a channel, but also, saving and fetching to flash memory is a channel, and so is sending over a fiber-optic link or the internet

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .