21

What is the Linux command-line command that can identify such files?

AFAIK the find command (or grep) can only match a specific string inside the text file. But I want to match whole contents, i.e. I want to see which files match regular expression \0+, ignoring the line end character(s). Maybe the find . cat | grep idiom could work, but I don't know how to make grep ignoring lines (and treat the file as binary).

Background: Every few days, when my laptop freezes, my btrfs partition looses information: files opened for write gets their contents replaced with zeroes (the size of the file remains more-or-less intact). I use synchronization and I don't want these fake files to propagate: I need a way to identify them so I can grab them from backup.

7
  • you mean files having numeric zeros in it ? Dec 20, 2012 at 11:08
  • 2
    I think it's about NULL characters rather than numeric zeroes.
    – gertvdijk
    Dec 20, 2012 at 11:10
  • 10
    Let's take a step back here. Every few days, when your laptop freezes? Why aren't we trying to fix that, the real problem here?
    – D_Bye
    Dec 20, 2012 at 11:32
  • 2
    @D_Bye that's good idea, but so far it didn't come too far:[unix.stackexchange.com/questions/57894/… Dec 20, 2012 at 13:23
  • 1
    have you considered the -v option to grep: filter out all files that have any byte 1 to 255. Jun 1, 2015 at 21:27

9 Answers 9

11

You can grep for ␀ characters using the Perl regex mode:

$ echo -ne "\0\0" > nul.bin
$ echo -ne "\0x\0" > non-nul.bin
$ grep -P "[^\0]" *.bin
Binary file non-nul.bin matches

So you can use this:

for path in *.foo
do
    grep -P "[^\0]" "$path" || echo "$path"
done
5
  • I get unexpected results, using GNU grep 2.5.4. Regardless of whether I use --binary-files=text or --binary-files=binary, it gives a true result for all non-empty data values, eg. "\0\0", "\0x\0", "abcd"... The exact code I used is: for typ in binary text ;do for dat in '\0\0' '\0x\0' 'abcd' '' ;do printf "$dat" >f; grep --binary-files=$typ -P '[^\0]' f >/dev/null && echo true || echo false; done; done
    – Peter.O
    Dec 21, 2012 at 3:37
  • 1
    I have now further tried GNU grep) 2.10. This later version does give the expected results... so, a belated +1
    – Peter.O
    Dec 21, 2012 at 3:44
  • 1
    Fails on a file created with printf '\0\n\0\0\n\n' > file or printf '\n' > file for that matters. Jun 1, 2015 at 21:42
  • 2
    @StéphaneChazelas OP did say "ignoring the line end character(s)." So any file consisting of only \0 and \n characters (even zero of either) would be a match.
    – l0b0
    Jun 1, 2015 at 22:59
  • 1
    Using grep is a good idea, because grep regular expression compiler is very fast. Using -P "[^\0]" to check for a non-nul byte seems to be the only possible way in grep. Good find! However, this grep call prints the names of files that contain non-nul bytes. OP asked for the inverse.
    – hagello
    May 21, 2021 at 14:13
7

I agree with what D_Bye says about finding the root of the problem.

Anyway to check if a file only contains \0 and/or \n you could use tr:

<file tr -d '\0\n' | wc -c

Which returns 0 for null/newline and empty files.

6
  • 2
    tr -d '\0\n' solves the newline issue, which then only leaves the issue(?) of empty files being listed in the output... It does process every byte of every file though (which may or may not be an issue) +1
    – Peter.O
    Dec 20, 2012 at 13:02
  • @Peter.O: I missed the newline requirement, thank you. This solution isn't very optimized and if it is to run on a lot of data it would be better with a solution that moves on upon finding non-matching bytes.
    – Thor
    Dec 20, 2012 at 16:39
  • It works very well. I my case I only had to make sure to exclude zero-length files. Thank you. Dec 20, 2012 at 19:21
  • 1
    This will also, however, count files with newlines in as being "empty".
    – Chris Down
    Dec 22, 2012 at 11:52
  • 1
    @ChrisDown: I made the answer text clear as to what it does. It's not clear what the OP wants to do with newline-only files.
    – Thor
    Dec 24, 2012 at 0:16
5

I suspect those files are sparse, that is they don't have any disk space allocated to them, they just specify a file size (du would report 0 for them).

In which case, with GNU find, you could do (assuming no file path contains newline characters):

find . -type f -size +0 -printf '%b:%p\n' | grep '^0:' | cut -d: -f2-
5
  • Good point. I never thought about it. I'll try. Using du will prevent from scratching contents of every single file in the file system, so the whole procedure wouldn't take 30+ minutes to complete. Dec 22, 2012 at 9:58
  • (and printf %b above reports what du would report) Dec 22, 2012 at 16:19
  • I would change -size +0 to -size +1 so zero length files are excluded from the results. Also files containing \n in their path will cause issues for this command.
    – Tyson
    Jan 21, 2020 at 1:38
  • @Tyson -size +0 is for sizes strictly greater than 0. -size +1 would be for sizes strictly greater than 512. The newline limitation was already mentioned. Jan 21, 2020 at 6:06
  • @StéphaneChazelas Thanks for enlightening me regarding -size +1, you are indeed correct. I've fixed my answer. :-)
    – Tyson
    Jan 22, 2020 at 7:47
5

Here's a small python program that can do it:

import sys

def only_contains_nulls(fobj, chunk_size=1024):
    first = True
    while True:
        data = fobj.read(chunk_size)
        if not data:
            if first:
                return 1  # No data
            else:
                return 0
        if data.strip("\0"):
            return 1
        first = False

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        sys.exit(only_contains_nulls(f))

And in action:

$ printf '\0\0\0' > file
$ ./onlynulls file && echo "Only nulls" || echo "Non-null characters"
Only nulls
$ printf a >> file
$ ./onlynulls file && echo "Only nulls" || echo "Non-null characters"
Non-null characters

You can check multiple files by using find's -exec, xargs, GNU parallel, and similar programs. Alternatively, this will print filenames that need to be dealt with:

files=( file1 file2 )
for file in "${files[@]}"; do
    ./onlynulls "$file" || printf '%s\n' "$file"
done

Bear in mind that if you're going to pass the output of this to another program, filenames can contain newlines, so you should delimit it differently (fittingly, with \0).

If you have a lot of files, it would be better to use an option for parallel processing, since this only reads one file at a time.

2
  • 2
    Beware, zero length files (eg: /etc/nologin, ~/.hushlogin, .nomedia, ...) are misidentified by this answer.
    – Tyson
    Jan 21, 2020 at 1:32
  • @Tyson Thanks for pointing that out! I've just fixed it.
    – Chris Down
    Jan 21, 2020 at 13:43
3

Find files which contain only null-chars '\0' and newline chars '\n'.
The q in sed causes each file search to quit immediately upon finding any non-null character in a line.

find -type f -name 'file-*' |
  while IFS= read -r file ;do 
      out=$(sed -n '1=; /^\x00\+$/d; i non-null
                      ; q' "$file")
      [[ $out == "1" ]] &&  echo "$file"
  done

Make test files

> file-empty
printf '%s\n' 'line1' 'line2' 'line3'      > file-with-text           
printf '%4s\n' '' '' xx | sed 's/ /\x00/g' > file-with-text-and-nulls
printf '%4s\n' '' '' '' | sed 's/ /\x00/g' > file-with-nulls-and-newlines
printf '%4s'   '' '' '' | sed 's/ /\x00/g' > file-with-nulls-only

output

./file-with-nulls-and-newlines
./file-with-nulls-only
1
  • Either the-print0 argument seems to be missing from find or the IFS= part is messed up. What was the intended delimiter?
    – Tyson
    Jan 21, 2020 at 1:44
3

This one-liner is the most efficient way to find 100% nul files using GNU find, xargs, and grep (assuming the latter is built with PCRE support):

find . -type f -size +0 -readable -print0 |
  LC_ALL=C xargs -r0 grep -LP "[^\x00]" --

The advantages of this method over other provided answers are:

  • non-sparse files are included in the search.
  • non-readable files aren't passed to grep, avoiding Permission denied warnings.
  • grep will stop reading data from files after finding any non-nul byte (LC_ALL=C is used to make sure each byte is interpreted as a character).
  • empty files (zero bytes) are not included in the results.
  • fewer grep processes efficiently check multiple files.
  • paths containing newlines or starting with - are handled correctly.
  • works on most embedded systems that lack Python/Perl.

Passing the -Z option to grep and using xargs -r0 ... allows further actions to be performed on the 100% nul files (eg: cleanup):

find . -type f -size +0 -readable -print0 |
  LC_ALL=C xargs -0 grep -ZLP "[^\x00]" -- |
  xargs -r0 rm --

I also recommend using the find options -P to avoid following symlinks, and -xdev to avoid traversing filesystems (eg: remote mounts, device trees, bind mounts, etc).

For ignoring the line end character(s), the following variant should work (though I don't think this is such a good idea):

find . -type f -size +0 -readable -print0 |
  LC_ALL=C xargs -r0 grep -LP "[^\x00\r\n]" --

Putting it all together, including removing the unwanted files (100% nul / newline characters) to prevent them from being backed up:

find -P . -xdev -type f -size +0 -readable -print0 |
  LC_ALL=C xargs -0 grep -ZLP "[^\x00\r\n]" -- |
  xargs -0 rm --

I don't recommend including empty files (zero bytes), they often exist for very specific purposes.

8
  • Being the fastest out of so many alternatives is a bold claim. I will mark your your answer as accepted if you add a benchmark :-) Jan 20, 2020 at 16:33
  • Such a benchmark would be dependent on many factors, including performance of the various disk subsystems.
    – Tyson
    Jan 21, 2020 at 0:16
  • Of course, but anything is better than nothing. Various approaches optimize CPU usage differently, so it makes sense to benchmark it on SSD or even on cached files. Take the machine you currently work on, write one sentence what it is (CPU type, no of cores, RAM, hard drive type), describe the file set (e.g. kernel source clone + 1GB file full of \0 with 900MB hole in it) and present timing of the results. If you do it in a way the benchmark is convincing for you, it will most likely be convicing for all of us Jan 21, 2020 at 9:40
  • "most embedded systems" don't have GNU utilities. More likely busybox ones. Jan 22, 2020 at 8:07
  • 1
    In [^\x00\n\r], the \n is redundant as grep works on the contents of lines which doesn't include the newline (\n) character. The grep -LP '[^\x00]' already ignores newlines. It finds files with no line containing non-NUL bytes, so would also report files containing only empty lines Jan 22, 2020 at 8:14
1

Here's a mix and abbreviation of the answers of @Tyson and @l0b0:

# --text: do not skip non-text files
grep --recursive --files-without-match -P '[^\0]'

If you want a minimum file size of one byte:

# -size +0c: more than zero bytes
find -type f -size +0 -exec grep --text --files-without-match -P '[^\0]' {} +
0

For using GNU sed you can use the -z option, which defines a line as zero-terminated strings and match for and delete empty lines like so:

if [ "$( sed -z '/^$/d' "$file" | head -c 1 | wc -c )" -eq 0 ]; then
    echo "$file contains only NULL!"
fi

The head command inbetween is just an optimization.

-1

Python

Single file

Define the alias:

alias is_binary="python -c 'import sys; sys.exit(not b\"\x00\" in open(sys.argv[1], \"rb\").read())'"

Test it:

$ is_binary /etc/hosts; echo $?
1
$ is_binary `which which`; echo $?
0

Multiple files

Find all binary files recursively:

IS_BINARY='import sys; sys.exit(not b"\x00" in open(sys.argv[1], "rb").read())'
find . -type f -exec bash -c "python -c '$IS_BINARY' {} && echo {}" \;

To find all non-binary files, change && with ||.

1
  • 1
    The question asked to identify files containing only nul characters (ignoring newlines), the Python code given here identifies files containing any nul characters.
    – Tyson
    Jan 21, 2020 at 1:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .