Quickly find all files that start with a given string

Question

I'm writing a command to find broken Cygwin symlinks.

Right now, I'm using:

egrep -rnw . -e "^\!<symlink>"

But this isn't very efficient. I only care if the file starts with !<symlink>, but grep will keep searching the rest of the file.

Is there any way I can efficiently search files by content prefix?

Answer 1

Building on the other answer and the comment on it,

find . -type f -exec awk '/^!<symlink>/ {print FILENAME} {nextfile}' {} +

Because it uses -exec … +, it will execute awk as few times as possible (based on the maximum argument list size).  If awk finds a line that matches the regular expression (i.e., the search string), it will print the filename.  (You don't need the backslash (\) if you put the string in single quotes.)  Obviously, if you also want to see the content of the line, you can say

{print FILENAME; print}

or

{print FILENAME ": " $0}

And then, unconditionally, it advances to the next file; so it effectively looks at only the first line of each file.  Note: nextfile does not seem to be defined by POSIX; it is probably only in GNU awk.

Answer 2

How about:

find . -type f -exec head -1v {} + | egrep -B 2 -nw -e "^\!<symlink>"

Answer 3

Also building on others ideas.

Have you thought of xargs? It can do some parallelization.

find . -type f -print0 | xargs -0 -P4 head -1 | egrep -wn "^\!<symlink>"

What it does: looks for all regular file and separate the list of files with '\0' instead of white space.
We instruct xargs to understand that '\0' are the separator and to run 4 concurrent processes (adjust that to your number of cores and disk speed). Xargs will run the head command on each file.
Finally grepping the results with your pattern.

Head will still be run plenty of time. But concurrently! Unless you need to do periodic search of this kind, I would not bother trying to optimize it further. The time it's taking to optimize more will not be gained by the faster runtime, if you run it only once anyway.