3

I'm writing a quick tool to inspect the contents of a node.js node_modules folder or python virtualenv for native dependencies. As a quick first approximation to this I wrote the following command.

find . | xargs file | awk '/C source/ {print $1} /ELF/ {print $1}'

I'm okay with false positives but not false negatives (e.g. files literally containing the string ELF or C source can be marked suspicious.), but this script also potentially breaks on long file names (because xargs will split them) and file names containing spaces (because awk will split on whitespace) and file names containing newlines (because find uses newlines to separate paths).

Is there a way to filter the paths generated by find by seeing if the output of file {} (possibly with some additional options to remove the path entirely from the output of file) matches a particular regular expression?

  • 1
    Use -exec cmd {} + instead of the nonstandard -print0. – schily Apr 6 '16 at 19:12
  • Your updated command looks better, but it is still susceptible to filenames with spaces or newlines. If you can get the list of files from find and iterate F over that list, then you can echo "$F" whenever the output of file $F | grep ... returns success. – Prem Apr 6 '16 at 19:22
  • 3
    OP - get rid of the pipe and do it find only; use file with -b so as to use grep with anchor e.g. find . -type f -exec sh -c 'file -b "$0" | grep -q "^ELF\|^C source" && printf %s\\n "$0"' {} \; – don_crissti Apr 6 '16 at 19:25
  • 1
    Visit this page; also follow the links there... – don_crissti Apr 6 '16 at 19:44
  • 1
2

The key factor in reaching find enlightenment ;) is:

find's business is evaluating expressions -- not locating files. Yes, find certainly locates files; but that's really just a side effect.

--Unix Power Tools

There is an alternate approach to this question that it's worth knowing about (as also described in Unix Power Tools, in the section "Using -exec to Create Custom Tests"):

find . -type f -exec sh -c 'file -b "$1" | grep -iqE "^ELF|^C source"' sh {} \; -print

It's worth knowing about this filtering method since it can be used for many more things than simply printing the name of the file; just change the -print operator to any other operator you like (including another -exec operator) and do what you like with it.


There is a performance drawback to this command (which is also present in the other answer), which is that since we are using \; and not +, we are spawning a shell for every single file. Using + to pass multiple files at once to the sh command and processing them with a for loop gives a noticeable performance advantage:

find . -exec sh -c 'for f do file -b "$f" | grep -qE "^ELF|^C source" && printf %s\\n "$f"; done' sh {} +

You can see the comparison for yourself by running both of the following commands and comparing the output of time:

time find . -exec sh -c 'for f do file -b "$f" | grep -qE "^ELF|^C source" && printf %s\\n "$f"; done' sh {} +
time find . -exec sh -c 'file -b "$1" | grep -qE "^ELF|^C source" && printf %s\\n "$1"' sh {} \;

The real point, though, is:

Never run a shell for loop on a list of files that is output from find. Instead, either run the action you need to do on each file directly within find by using the -exec operator, or embed a shell for loop within a find command and do it that way.

Some additional reasons:

  • Does + in place of \; affect the character used to separate lines of output from find or whether a subprocess is spawned for each invocation? – Gregory Nisbet Apr 6 '16 at 23:24
  • @GregoryNisbet, the latter—see POSIX specs for find (or just look at man find). + affects the behavior of the -exec operator and makes find run whatever command is specified after -exec with multiple files as arguments, rather than just one. Actually, I just realized your answer has no performance benefit as written; I will update my answer. – Wildcard Apr 6 '16 at 23:28
  • @StéphaneChazelas, I'm trying to find how for f; do is nonstandard and from the line about for_clause on this page it seems it should be standard...sequential_sep can be a ;, right? I'm not sure I've got the hang of the grammar depiction format, though.... – Wildcard Nov 8 '16 at 0:00
  • 1
  • So, with the 2016 edition now released, that's no longer true, I'll remove my comment. (you still need for f do for the Bourne shell though). – Stéphane Chazelas Nov 8 '16 at 12:38
3

It is easiest to execute a small script for every file that checks the brief-mode output of file and prints the path if the output of file matches ELF or C source, the path is passed in as $0.

find . -type f -exec sh -c \
    'file -b "$0" | grep -q "^ELF\|^C source" && printf %s\\n "$0"' {} \;

This solution has the following advantages over the original

-type f filters out directories immediately instead of relying on the output of file

Passing in the argument as {} avoids issues related to whitespace or newlines in the file name.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.