3

I have a very strange situation where I have a file system that contains a number of files that are of varying size but appear to have been corrupted and show no contents when cat-ing the file.

I have no idea how many of these files there are and they look like all the other files around them. So far the best method I've found to discover them is to run find . -type f | xargs wc -l and look for the files that return 0.

However, some of these files are 10s of GB and even when they're empty take over a minute to check one file. I have to check approximately 50 million files, so that's pretty much a non-starter.

My biggest question is if there's a lower level way to check the contents of these files that will be much faster than running a simple find.

  • 1
    No output from cat does not mean that the file is empty. There are reliable methods such as find or stat or shell features to find empty files. – countermode Nov 15 '16 at 15:56
  • I tried that, these files don't show up using the empty flag because they still have some sort of contents. It's just not the content that is expected. When trying to open these files with less, it thinks they are a binary file and shows garbled content, but these are (supposed to be) regular text files. – Kevin L Nov 15 '16 at 16:01
  • @countermode - could you please elaborate further on these features you mention? I know these files aren't empty, but they no longer contain their original data in a useful format. – Kevin L Nov 15 '16 at 18:08
4

and show no contents when cat-ing the file.

No output of cat for a file does not imply that the file is empty as the following experiment shows:

$ truncate -s 1M foo    
$ ll foo
-rw-r----- 1 user users 1048576 Nov 15 19:28 foo
$ file foo
foo: data
$ cat foo
$ 

cat does output 1MiB worth of NUL characters, but those characters happen to be invisible in a terminal.

Your question about empty files is somewhat ambiguous. Does foo above qualify? If "empty" means length zero, then find could aid you:

find dir -type f -empty

lists all zero length files under dir. If your find doesn't support -empty, you can use -size 0 instead.

In shell scripts you may use the -s file expression which is true if file exists and is non-empty. In order to check whether a file is empty (given that it exists) use something like

if [ \! -s file ] ; then ... ; fi

Alternatively, you may use stat(1). Here with the GNU variant:

$ stat --format=%s foo
1048576

which you may use further in comparisons.

4

From your clues:

  • they are or non-0 size, as reported by ls -l
  • that cating them appears to show nothing
  • wc -l returns 0.

We can tell:

  • they contain no newline character (wc -l counts the newline characters)
  • if they contain any character, they are invisible in a terminal

While there are a great number of characters that are invisible in a terminal like most control characters, and some extended unicode ones, for many different corrupt files to show that behaviour makes me think that's likely to be the NUL character.

A corrupt file can be seen as all-zeros if all references to data blocks have been removed to it, with the size attribute in the inode left intact. That's fully sparse files.

Unless the block count field in the inode is also corrupted, you may be able to detect those with (assuming GNU find and awk):

find . -size +0 -printf '%b%p\0' | awk -v RS='\0' '
  /^0/{print substr($0, 2)}'

That is, find files whose size is non-zero but disk usage is null.

0

My biggest question is if there's a lower level way to check the contents of these files that will be much faster than running a simple find.

Try du:

$ truncate -s 4G my4g
$ ls -l my4g
-rw-rw-r-- 1 tange tange 4294967296 Mar  4 15:34 my4g
$ cat my4g
$ du my4g
0       my4g

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.