3

I don't have a problem with getting the name of the file where grep finds a match, however, something in the file hierarchy I am searching through takes forever (hours) to search (it is 1,4 GB data in 2000 files on an SSD so it shouldn't take that long) and I want to know what. Therefore, I would like some kind of verbose output where grep lists every file as it processes them.

How can this be done?

Using BSD grep on a Mac.

2
  • 5
    Not with grep -r directly. But with find ... -exec ... {} + you can do whatever you want.
    – user313992
    Commented Oct 26, 2020 at 22:28
  • If grep -v meant --verbose, this is what we might expect it to do (but grep -v means "invert the match"). This would be a worthy enhance for Gnugrep --verbose
    – smci
    Commented Sep 21, 2022 at 16:36

3 Answers 3

5

You have to call separate grep processes, together with printing the filename before starting to search. Of course this will be slower but in your case, as grep has a tiny starting load time, it will not disturb you, comparing to the current situation.

Using find and exec:

find . -type f -exec sh -c 'printf "checking: %s\n" "$1"; grep pattern "$1"' sh {} \;

Or find and mass exec, to call one shell to loop

find . -type f -exec sh -c 'for f do printf "checking: %s\n" "$f"; grep pattern "$f"; done' sh {} +

Or using find and xargs, with GNU zero separation for the file arguments:

find . -type f -print0 | xargs -0 sh -c 'for f do printf "checking: %s\n" "$f"; grep pattern "$f"; done'

By the way, as I see you are on macOS, please consider to install GNU software, for grep, sed, awk, etc.

6
  • -d is GNU specific and a lot less portable than -0 (and would cause problems with filenames containing newlines (IIRC MacOS/X used to ship with such files in /etc)). Portably, you need to give at least one file to find (so here find . ... if you want to find files in the current working directory). Commented Oct 27, 2020 at 6:54
  • Well, yes, I omit the -d version, I thought -d was not GNU specific and added as an xargs alternative for mac (although not considering newlines, but for the case -0 is not a choice). I hope -exec is fine for mac. Also I 'm surprised that macOS had newlines in default filenames, while having no default zero separation in their shell.
    – thanasisp
    Commented Oct 27, 2020 at 7:27
  • -exec is standard, -0 is not but should be fine too. -r is less portable that -0, but FreeBSD supports it (it's the default there though which makes it not POSIX compliant, so I expect macOS' differs there as it's been certified as compliant). Commented Oct 27, 2020 at 7:30
  • In your first example, where do I enter the pattern ($1)? You seem to use a variable.
    – d-b
    Commented Oct 28, 2020 at 8:40
  • No, you don't. $1 is the argument passed from find -exec into the shell sh we execute. Also this way we can do various things onto the argument, e.g. apply parameter expansion etc. You just have to define the parameters of your find command, the starting directory (. to mean the current one) what files you select, by type, name, etc.
    – thanasisp
    Commented Oct 28, 2020 at 8:43
3

This can be done at a point in time with lsof, even on an already-running grep, though it won't continuously print the files as it goes.

First get the PID of the running grep process:

$ pgrep -l grep
21531 grep

Then list every open file of that process:

$ lsof -p 21531
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
grep    21531 izkata  cwd    DIR  253,1     4096 22020097 /var
grep    21531 izkata  rtd    DIR  253,1     4096        2 /
grep    21531 izkata  txt    REG  253,1   219456 22544390 /bin/grep
grep    21531 izkata  mem    REG  253,1  3004224 11805539 /usr/lib/locale/locale-archive
grep    21531 izkata  mem    REG  253,1   144976 29104688 /lib/x86_64-linux-gnu/libpthread-2.27.so
grep    21531 izkata  mem    REG  253,1  2030544 29104673 /lib/x86_64-linux-gnu/libc-2.27.so
grep    21531 izkata  mem    REG  253,1    14560 29104676 /lib/x86_64-linux-gnu/libdl-2.27.so
grep    21531 izkata  mem    REG  253,1   464824 29103098 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
grep    21531 izkata  mem    REG  253,1   170960 29104669 /lib/x86_64-linux-gnu/ld-2.27.so
grep    21531 izkata  mem    REG  253,1    26376 12064533 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
grep    21531 izkata    0u   CHR  136,8      0t0       11 /dev/pts/8
grep    21531 izkata    1u   CHR  136,8      0t0       11 /dev/pts/8
grep    21531 izkata    2u   CHR  136,8      0t0       11 /dev/pts/8
grep    21531 izkata    3r   DIR  253,1     4096 22020408 /var/lib
grep    21531 izkata    4r   REG  253,1    35765 22156714 /var/lib/dpkg/info/systemd.md5sums
grep    21531 izkata    5r   DIR  253,1     4096 22020692 /var/lib/dpkg
grep    21531 izkata    6r   DIR  253,1   471040 22021003 /var/lib/dpkg/info

Most of it is just things grep needs to run, but those last 4 lines are directories and the file grep is currently reading (/var/lib/dpkg/info/systemd.md5sums in this example).

4
  • 1
    See also lsof -ad3-999 -p 21531 to only list the files opened on fds above 2 to trim that output a bit. Commented Oct 27, 2020 at 14:24
  • @StéphaneChazelas FDS?
    – d-b
    Commented Oct 28, 2020 at 0:08
  • @d-b file descriptors Commented Oct 28, 2020 at 1:33
  • Your suggestion failed, grep was till open but it wasn't reading any file (other than its own files). It seems it "gave up" on something and was just hanging there.
    – d-b
    Commented Oct 28, 2020 at 8:45
3

With zsh (which I understand is now the default interactive shell on macOS), you could do:

for f (**/*(ND-.)) (set -x; grep pattern $f)

(where set -x, short for set -o xtrace prints the command that is being executed on stderr like in all POSIX shells, and **/*(ND-.) looks for all regular files after symlink resolution recursively, including hidden ones).

With GNU or FreeBSD xargs (so hopefully macOS' as well), you could do something similar with:

find . -type f -print0 | xargs -0tn1 grep pattern

(where the -t option does the equivalent of xtrace)

That doesn't look into symlinks to regular files though (GNU find has -xtype f for that, but not FreeBSD's). Whether grep -r itself looks into symlinks to regular files depends on the grep implementation and version, you may want to double-check with yours.

Those run one grep per file, so is going to less efficient than a grep -r.

Another approach wouldbe to use a system call tracer like strace/truss/dtruss/tusc or whatever the equivalent is on your macOS and trace the open() or openat() or whatever system call grep uses to open the files.

On GNU/Linux, that would be:

strace -e open,openat grep -r pattern .

I don't know what the equivalent would be on macOS.

5
  • the -r for xargs failed. What is it supposed to do?
    – d-b
    Commented Oct 28, 2020 at 8:42
  • @d-b, it avoids running the command if the input is empty. You can omit it in this case as it should be harmless here if grep is called without argument as it would then look in its stdin which would be empty (either /dev/null or the pipe depending on the xargs implementation). Commented Oct 28, 2020 at 9:15
  • Thank you. The result of this command was interesting, everything worked fine! There seems to be some issue with grep -ir itself that disappears when feeding grep with full paths instead. I therefore tried to remove -print0 since I am no longer interested in the log but then I got an error message xargs: insufficient space for argument?
    – d-b
    Commented Oct 28, 2020 at 9:50
  • Next, I removed xargs too, and just ran find . -type f | grep PATTERN but then it finished very fast and no results were returned (when running the whole command as suggested in your post I get a few results). What am I missing here?
    – d-b
    Commented Oct 28, 2020 at 9:53
  • 1
    @d-b If you don't want the log, it's -t you want to remove, not -print0. If you remove xargs, you're grepping the output of find, not the contents of the files whose path is printed by find. Commented Oct 28, 2020 at 10:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .