23

I wrote a function that checks for a corrupted archive using a CRC checksum.

To test it, I just opened the archive and scrambled the content with a hex editor. The problem is that I do not believe that this is the correct way to generate a corrupted file.

Is there any other way to create a "controlled corruption", so it won't be totally random but can simulate what happens with real corrupted archives? I never had to corrupt something on purpose so I am not really sure how to do so, beside the random scrambling of data in a file.

  • What tool are using to "archive", by corrupt to do you mean the contents of one of the files in the archive, or the archive itself? – Drav Sloan Aug 10 '15 at 18:30
  • I am using tar as archive format. I would like to corrupt only the content of the file; so the archive itself is still recognized as tar file. My function extract the file; I have a case where there file is corrupted, but I want to check what happens when the file inside the archive is corrupted. – rataplan Aug 10 '15 at 19:05
22

I haven't done much fuzz testing either, but here's two ideas:

Write some zeroes into the middle of the file. Use dd with conv=notrunc. This writes a single byte (block-size=1 count=1):

dd if=/dev/zero of=file_to_fuzz.zip bs=1 count=1 seek=N conv=notrunc

Using /dev/urandom as a source is also an option.

Alternatively, punch multiple-of-4k holes with fallocate --punch-hole. You could even fallocate --collapse-range to cut out a page without leaving a zero-filled hole. (This will change the file size).

A download resumed at the wrong place would match the --collapse-range scenario. An incomplete torrent will match the punch-hole scenario. (Sparse file or pre-allocated extents, either read as zero anywhere that hasn't been written yet.)

Bad RAM (in the system you downloaded the file from) can cause corruption, and optical drives can also corrupt files (their ECC isn't always strong enough to recover perfectly from scratches or fading of the dye).

DVD sectors (ECC blocks) are 2048B, but single byte or even single-bit errors can happen. Some drives will probably give you the bad uncorrectable data instead of a read-error for the sector, especially if you read in raw mode, or w/e it's called.

  • 1
    Because of how hard drives work, zero-filling on a 4K-aligned 4K block, or a 512-byte-aligned 512-byte block, is the most realistic. – Mark Aug 11 '15 at 1:20
  • @Mark: Oh, if you're thinking about HD-induced corruption, yes. Bad RAM in someone's computer can flip a bit in the middle of a file. Similarly, a round trip to/from a bad optical disc can zero a smaller chunk (DVD ECC codes work on a different chunk size). – Peter Cordes Aug 11 '15 at 4:06
10

The other answers seems mostly concerned with hardware errors. Let me list some software-caused corruptions:

  • LF replaced with CRLF.
  • CR removed. (Even if not followed by LF)
  • Extra Null bytes inserted.
  • Extra Unicode "Byte Order Mark" inserted.
  • Character set converted from UTF-8 to Latin-1 or vice versa.
  • DOS EOF-character(#1A) deleted, even when not at End Of File.

These things are fairly harmless when happening to text files, but generally deadly when applied to binary files.

7

Use dd to truncate the file, or try a binary editor like hexer to edit and introduce some corruptions.

Example of truncating file using dd

Create 5MB file

# dd if=/dev/zero of=foo bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.0243189 s, 216 MB/s
# ls -l foo
-rw-r--r-- 1 root root 5242880 Aug 12 20:13 foo
#

Truncate 10 bytes off the end

# dd if=foo of=foo-corrupted bs=1 count=5242870
5242870+0 records in
5242870+0 records out
5242870 bytes (5.2 MB) copied, 23.7826 s, 220 kB/s
# ls -l foo foo-corrupted
-rw-r--r-- 1 root root 5242880 Aug 12 20:13 foo
-rw-r--r-- 1 root root 5242870 Aug 12 20:14 foo-corrupted
#

Hexer man page

HEXER(1)                              General Commands Manual                             HEXER(1)

NAME
   hexer - binary file editor

SYNOPSIS
   hexer [options] [file [...]]

DESCRIPTION
   hexer  is  a  multi-buffer  editor  for  viewing  and  manipulating binary files.  It can't
   (shouldn't) be used for editing block devices, because it tries to load the whole file into
   a  buffer (it should work for diskettes).  The most important features of hexer are:  multi
   buffers, multi level undo, command line editing with completion, binary regular expressions
   (see  below).   The  user  interface  is  kept similar to vi, so if you know how to use vi,
   you'll get started easily.
  • Thanks Steve. would this simulate what happens in a real case scenario? Like you are copying an archive from network and it gets corrupted? I believe that an unsuccessful download can be simulated with dd, to truncate the file. Would that be accurate? – rataplan Aug 10 '15 at 19:07
  • 2
    Yes, by truncating the file using dd, that would simulate a real world scenario where only part of the file gets created. And editing using hexer to introduce some bogus content would simulate another type of corruption. As an aside the md5sum may be worth looking at, it computes md5 checksum for a file. – steve Aug 10 '15 at 20:58
  • 1
    @newbiez, truncating randomly simulates a network failure, while truncating on a 4Kb or 512-byte boundary simulates a disk failure. – Mark Aug 11 '15 at 1:21
  • how do you actually truncate file using dd? – Edward Torvalds Aug 12 '15 at 13:54
  • @edward torvalds - dd truncate example added – steve Aug 12 '15 at 19:15
2

Suggestion:

Start writing to an archive and stop the thing doing the writing before it finishes. This can occur during power cuts and other scenarios.

Real life scenario:

I once corrupted a zip file by trying to copy more data into it than would fit on the medium. Windows (this was Windows 7 in safe mode ftr) tried to complete the action before figuring out if there was enough space, and by the time it had figured it out the file was half-complete and thus corrupt. I hope they fixed that issue in later versions of windows or that was just a safe mode thing.

2

Another common type of corruption is bit-twiddling: where a single bit (or multiple bits) get toggled in a datastream.

So a byte 1111 0000 might become, say, 1111 0010 or 1011 0000 or 1110 1100 or whatever.

Parity and count-the-ones checksumming systems have problems with things like 1110 1000 where there are an equal number of sets and unsets, since both the parity and the number of ones remain the same.

So replacing all instances of a random character with its inverse, say 0x57 to 0x75 ('9' to 'K') or vice versa might not be detectable. For systems which have mysql, the command "replace" exists for just such a purpose:

replace K 9 < goodInputFile > corruptedOutputFile

You can also try swapping the letter K and 9 around, which will be a particularly good test if they both appear the same number of times in the file:

replace K 9 9 K < goodInputFile > corruptedOutputFile

Use man replace for more info.

0

Random changes to corrupt test data aren't a good approach, since you cannot reproduce the sample to re-run the tests.

I would be happy with only 3 samples, changing just 1 bit in the first byte, in the last byte and in any middle byte. But just 1 bit, not the whole byte.

But the best test sample would be one where you could generate samples changing each one bit of the file from the first to last byte. This cannot be (usually) got with usual tools, you need to build one (I guess).

With this approach you isolate a lot of possibilities including endianess if your algorithm is based in one kind of endianess. In other hands big sample can consume a lot of time to process.

At last, some sample truncating or adding bytes will complete your tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.