I have some SATA/IDE hard drive enclosures that I use for backups. The drives for formatted with VFAT, since I need to be able to share them with Windows and OS/X machines.

The backup scripts pipe the output through vdmfec to provide error correction for the files. (I started doing this when I found that some old backup files were damaged.)

The example on the manpage refers to 1.44MB "floppy" discs.

So my question is: is it worth using vdmfec for hard drives (as I've heard that newer ones have built-in error correction)? And if so, should I be using different settings than the defaults?

Also: are there better tools for Linux than vdmfec for error correction?

  • Another similar tool: dvdisaster.net/en – Wodin Apr 10 '11 at 14:08
  • @Wodin Thanks for the ref to another tool, but I don't use DVDs/CDs. (They're just not big enough for my backups.) – Rob Apr 10 '11 at 15:43

Adding Error Correcting Codes (ECC) to your backups is definitely a good thing. You trade off some amount of extra space to gain some robustness in the form of data redundancy.

vdmfec looks like a nice little tool. By default, it assumes a block size of b=1024 bytes, and it writes out N=18 blocks for every K=14 blocks of input. This means that by default, the output size is inflated by N-K=4 blocks for every 14 blocks, for a size increase of about 29%. You can play with the parameters K and N to get more or less redundancy, although my feeling is that the defaults are probably okay for most uses.

You need to be aware of a couple of potential gotchas, though. First, according to the man page:

Note that the N, K, and blocksize parameters are NOT written to the output! You must specify the same parameters when you run the decoder. (Actually, the decoder is capable of explicitly detecting an invalid K value, but incorrect blocksize or N values will result in bad blocks and decode failure.)

So remember to document the particular parameters you use for each of your backups. Even if you always use the defaults, it may be a good idea to explicitly document those values, since defaults can change in newer versions of the program.

Secondly, the man page says this:

The decoder is capable of reading from non-seekable media such as pipes, however, buffer underruns are not detected and will result in failure. Also, when reading from a pipe the entire file must be read. Reading from a seek-able stream can be faster because only K good blocks out of N need to be read.

This means that it is best (and safest) to put vdmfec last in your command pipeline when creating your backups. E.g.:

tar -cf - /data | gzip | vdmfec > /backups/data.tgz.vdm

By doing the vdmfec encoding last when backing-up, it means you can use the vdmfec decoder on a seekable file instead of on a non-seekable pipe when restoring, and thus avoid the potential problems that can arise when using pipes. E.g.:

vdmfec_decode /backups/data.tgz.vdm | zcat | tar -xf -

In summary, I would say that vdmfec is a good tool to use to make your backups more robust against data loss due to errors caused by media degradation. Today's hard drives are using ever higher bit densities to achieve their ever increasing multi-terabyte capacities, so I think it is definitely worth adding some extra redundancy to backups, just for safety.

And finally: Another tool that has a similar function to vdmfec is par2 (man page), but it works in an entirely different way. par2 may be a more suitable tool if you find you need to split your archive files, since par2 was developed to add ECC to large files split into numerous smaller files (e.g. for posting to USENET binaries groups).

  • I pipe the output of vdmfec to split (since files on VFAT are limited to 4GB, and my backups are significantly larger). – Rob Apr 12 '11 at 19:42
  • 1
    @Rob: In that case, I recommend you look into par2. Briefly, par2 works by running as a post-processing step on your large archive files. It creates a number of files containing redundancy codes that can be used to repair the original files, should they be corrupted. The number and size of the redundancy files is configurable. – Steven Monday Apr 12 '11 at 20:29
  • @Steven I don't see what I get from using par2 that I already don't have with vdmfec. – Rob Apr 13 '11 at 1:58
  • I've also found another tool, zfec pypi.python.org/pypi/zfec though I think it uses the same algorithm as vdmfec – Rob Apr 13 '11 at 1:59
  • @Rob: It was just a suggestion. If vdmfec works for you, then keep using it! Just be certain that you are able to restore from your backups, when/if the time comes. – Steven Monday Apr 13 '11 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.