2

I have received a tar archive from a co-worker and after decompresing I found that some text files are corrupted. More precisely they are filled with zeros. They have correct size but all bytes are equal to 0x00.

Can this situation be caused by some uncompatibility of tar versions or eg. chinese character contained in the files or the files had to been corrupted in the time of compression? I do not expect that there was a problem during transfer because control checksum is OK.

4
  • @goldilocks, 16-bit encodings contain plenty of zero bytes. (See my answer.)
    – alexis
    Commented Mar 4, 2013 at 14:04
  • @alexis : Alright, I had thought UTF-16 always used the high bit as a marker for bytes 2-4, but evidently the BMP characters are a single 16-bit value and the top half of that would often be zero. Sorry!
    – goldilocks
    Commented Mar 4, 2013 at 14:37
  • 1
    Does tar tf the-archive.tar | sort | uniq -d report anything by any chance? (what about tar tvf the-archive.tar | grep -v '^[ld-]')? What the operating system and tar version and file system where the tar file was made and where it is extracted? Commented Mar 4, 2013 at 15:36
  • The archive was created somewhere in China and I extracted in Czech Rep. in Ubuntu 12.10. Android sources was archived and some makefiles are broken as I described...
    – Honza
    Commented Mar 4, 2013 at 18:18

2 Answers 2

2

Are you sure all bytes are equal to 0x00? In that case your files contain no information at all (except for their size). No program could possibly store or transmit information as all zeros (unless it's telepathic).

What can happen is that a file has alternating text and zero bytes. Here's what this would mean: You've received files that contain unicode text, encoded as UTF-16 (or a near-equivalent). Each character takes up 16 bits (two bytes). Unicode assigns English letters and symbols to their ASCII character code, which means that, for example, letter A is hex 41 in ASCII and 00 41 in Unicode. The result is that if you write out "Hello" as UTF-16 and read it in as 8-bit text, you'll see this:

\0 H \0 e \0 l \0 l \0 o

In that case it wouldn't be the fault of tar. But if you really received all-zero files and the checksums check, there is definitely something wrong with the creating program. Not a version problem, but who knows? Perhaps a hardware problem causing the generating program to read all zeros.

(It's also possible, of course, that the files were correctly archived, and the bug is in the program that created the archived files).

5
  • Thank you for answer, but they are surely zero. There must be a problem on my coworker site...
    – Honza
    Commented Mar 4, 2013 at 14:10
  • Ah, I'd say it definitely must be. Good luck with it then.
    – alexis
    Commented Mar 4, 2013 at 14:12
  • 1
    That's why it's called the Telepathic ARchiver ... or short TAR ... it ought to know! Commented Mar 4, 2013 at 14:56
  • @0xC0000022L In that case I'd prefer it simply stored the number of zero bytes. Would cut down drastically on backup sizes.
    – user
    Commented Mar 4, 2013 at 15:08
  • 1
    @MichaelKjörling Unix philosophy: tar produces long streams of 0 bytes, gzip transforms them into a count. Commented Mar 4, 2013 at 23:10
1

The most likely problem is that the tar got corrupted while it was being created. Due to the way the tar format is defined (since it is intended to be a streaming archiver) it must determine the file length ahead of time. It records this length in the tar header, then starts writing the file contents to the tar file. If for some reason there is an error reading the file, or if the file shrinks while it is being archived, it will fill in NULLs. This is needed so that the length specified in the header is still valid upon extraction (it can't go back and modify the header due to its streaming nature, and if it didn't pad the file with NULLs that would cause an error when extracting the next file in the archive).

Also, since tar deals with binary data (it has no "text" mode), there shouldn't be any issue (as far as tar is concerned) with different language encodings.

1
  • I reproduced a similar issue with an SD card on OSX. cp refused to copy the files from the SD card (citing an I/O error), but tar happily appeared to tar the files. I guess tar is too fault tolerant and created the tar file, even though it was useless, since all extracted files were pure 0, confirmed with od -t x1 files*. Commented Jan 12, 2018 at 0:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .