How to safely use the output of grep in a script?

Question

In a script, I want to find files that contain some text. I need to know the file the text is found in, and the full line within the file that the text is found in. grep is the utility that does this, but how can I get the output into a usable form, given that there can be : in filenames? Is there some sort of --porcelain mode for grep that I can use, kinda like git commands often have?

Example: I have a folder full of files named like test-num:1:date:jan-2 that I want to grep through. The files contain FAILURE:<some reason> or SUCCESS:<some reason> (among other stuff). I need a script that searches for certain reasons and stores the name of the file, and the reason (the whole line of text is fine) for later processing. The output can be in any sort of data structure, as long as I can run code over it.

Answer 1

There is no such thing as a grep --porcelain, handling special characters in filenames has always been an afterthought in UNIX. You could try something like this, at the price of efficiency:

pattern='some pattern'
for file in ./*; do
    grep -- "$pattern" "$file" | while read -r line; do
        printf 'file: %s, line: %s\n' "$file" "$line"
    done
done

Answer 2

Recent(-ish) versions of GNU grep have an option -Z which makes the output unambiguous, but it's mostly aimed at uses like grep -lZ … | xargs -0. It still works if you're listing line contents, the null byte replaces the colon and the line content still ends in a newline¹, but shells aren't good at dealing with null bytes, so you'll have a hard time parsing this output.

One simple solution (with a slight performance penalty) is to run grep on each file individually.

Another solution is to use a language like Perl or Python. Perl is pretty good at emulating grep; grep REGEX is basically perl -ne '/REGEXP/ and print'.

But you may not need this at all if the output isn't actually ambiguous. For example, if the matching lines don't contain colons then the file name is everything on a line up to the last colon. If the matching lines all start with SUCCESS or FAILURE and these words don't appear in file names then you can use this to locate the separation, etc.

¹ _{Except when using -z to filter null-terminated records rather than newline-terminated records, then null is both the filename terminator and the result terminator; without -o the output is still unambiguous, with alternating output records being file names and matching records in the output.}

Answer 3

How to safely use the output of grep in a script?

... The output can be in any sort of data structure, as long as I can run code over it.

Shell scripts don't really have data structures. There are arrays, but that's about it—and it's not easy to get piped output into an array safely. (Filenames can contain newlines.)

The best way to run code over your files in a shell script is to just run the code over the files—not try to save the filenames for later use.

To do this, use find:

find somedir -type f -exec grep -q somepattern {} \; -exec somecommand {} \;

---

However, from reading your question more closely, it looks like you don't actually want to run code over your files, you just want to do some text processing on certain lines. In this case the GNU Grep option -z is probably what you want. That, and a knowledge of Sed or Awk, will handle your question.

---

It might be smart to change your file naming convention.