My problem is that I need to:

  1. Find all lines matching regex_pattern in all files (deep search) in a given root directory
  2. For each line that matches, output:
    • File name
    • Line number containing the match
    • Line contents
    • The regex pattern
  3. Import the above data into Excel (so a CSV, or delimited output format comes to mind)

What is the easiest way to do this task?

For the sake of proving I thought about this somewhat, I would write a Perl script that took as input a single fully qualified filename and a regex pattern, and process the lines using the approximate Perl below (I haven't tried yet, but this is what my first attempt would resemble):

while (<FILE>) {

  if ($_ =~ m/regex_pattern/) {
    # output: file_name\tline_number\tregex_pattern\t$_
    # ignore escaping issues for the time being

I'm still not sure how I'd pass in the contents of each directory with a recursive search into this Perl script. I can do the searching in Perl but I'm sure there's a nice Unix/Linux way to do this.

I'm not married to Perl. If there's a way to do it chaining together standard Unix/Linux tools, that'd be awesome. If not, I prefer to use Perl as I'm somewhat familiar with it's syntax.

  • 1
    "For each line that matches, I need the following outputted" – but not the file. Interesting... – Hauke Laging Jun 4 '13 at 1:48
  • @HaukeLaging Good call, thanks for pointing that out – jglouie Jun 4 '13 at 1:48

Something like this?

find /search/root -type f -exec awk 'BEGIN{pattern="regex_pattern"} $0 ~ pattern {printf "%s,%s,%s,%s\n",FILENAME,FNR,$0,pattern}'  {} +
  • Worked like a champ... very elegant. Thank you :) – jglouie Jun 4 '13 at 2:23
start cmd:> find . -type f -name 'search*' -exec awk -v regex=foo \
cont. cmd:>   '$0 ~ regex {print FILENAME,FNR,regex,$0 }' {} +
./searchfile1 1 foo a_foo_b
./searchfile2 1 foo foo

In Perl, taking advantage of the null filehandle which operates on command line arguments:

#!/usr/bin/perl -n
$, = "\t";  # separator added between arguments to print
while (<>) {
  if (/regex_pattern/) {
    # $ARGV contains the current file name, $. contains the current line number,
    # $_ contains the current line including its terminating newline
    print $ARGV, $., 'regex_pattern', $_;
  $. = 0 if eof;  # reset the line number between files

To pass the file names to the Perl script, in ksh93 or bash ≥4 or zsh, you can use the ** pattern to traverse subdirectories recursively. In ksh, you need to enable this pattern first with set -o globstar. In bash, you need to enable it with shopt -s globstar.

shopt -s globstar
name_of_perl_script **/*

If your shell doesn't have **, or if you run into a “command line too long” error, you can use find instead.

find . -type f -exec name_of_perl_script {} +

You can also do this by combining more specialized tools. You probably already know grep to search a pattern in files. The -n option makes it print the number of each matching line. Passing /dev/null as a file name is a trick to make sure that grep also prints the file name (it won't do it if there happens to be a single file on its command line).

grep -n 'regex_pattern' **/*

All that's missing is changing the separators if necessary (grep inserts : between the file name, the line number and the line content), and inserting the regex at the right place. This simple replacement job is a perfect fit for sed. Be sure to quote the regex properly.

find . -type f -exec grep -n 'regex_pattern' {} + |
sed 's/^\([^:]*\)\([^:]*\)/\1\t\2\tregex_pattern\t/'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.