How can I cat the contents of files found using find into a single file?

Question

I managed to shoot myself where it hurts (really bad) by reformatting a partition that held valuable data. Of course it was not intentional, but it happened.

However, I managed to use testdisk and photorec to recover most of the data. So now I have all that data distributed over almost 25,000 directories. Most of the files are .txt files, while the rest are image files. There are more than 300 .txt files in each directory.

I can grep or use find to extract certain strings from the .txt files and output them to a file. For example, here's a line that I've used to verify that my data is in the recovered files:

find ./recup*/ -name '*.txt' -print | xargs grep -i "searchPattern"

I can output "searchPattern" to a file, but that just gives me that pattern. Here's what I really would like to accomplish:

Go through all the files and look for a specific string. If that string is found in a file, cat ALL the contents of that file to an output file. If the pattern is found in more than one file, append the contents of subsequent files to that output file. Note that I just don't want to output the pattern I'm searching for, but ALL the contents of the file in which the patterns is found.

I think this is doable, but I just don't know how to grab all the contents of a file after grepping a specific pattern from it.

Answer 1

If I understand your goal correctly, the following will do what you want:

find ./recup*/ -name '*.txt' -exec grep -qi "searchPattern" {} \; -exec cat {} \; > outputfile.txt

This will look for all *.txt files in ./recup*/, test each one for searchPattern, if it matches it'll cat the file. The output of all cated files will be directed into outputfile.txt.

Repeat for each pattern and output file.

---

If you have a very large number of directories matching ./recup*, you might end up with a argument list too long error. The simple way around this is to do something like this instead:

find ./ -mindepth 2 -path './recup*.txt' -exec grep -qi "searchPattern" {} \; -exec cat {} \; > outputfile.txt

This will match the full path. So ./recup01234/foo/bar.txt will be matched. The -mindepth 2 is so that it won't match ./recup.txt, or ./recup0.txt.

Answer 2

Rather than outputting your pattern, output the filename using "-l" on grep, and then use that as input to cat.

find ./recup*/ -name '*.txt' -print | xargs grep -li "searchPattern" | xargs cat

or

cat $( find ./recup*/ -name '*.txt' -print | xargs grep -li "searchPattern")

I suspect that you can fill in the remaining details. BTW, if you may have spaces or other odd characters in the filenames (unlikely in this specific case, but for future purposes), use -print0 on the find and -Z on the grep, combined with the -0 option on xargs to use null bytes between filenames rather than newlines.

find ./recup*/ -name '*.txt' -print0 | xargs -0 grep -Zli "searchPattern" | xargs -0 cat

Answer 3

This is not exactly optimal code, but it's very straightforward and will work fine if efficiency isn't an issue. The problem is that it'll grep through the files multiple times, even if the string has already been found in them.

Firstly, search for your strings and write the matching files to a list.

find ./recup*/ -name '*.txt' -execdir grep -il "searchPattern" {} >> /tmp/file_list \;

Repeat this step replacing searchPattern as necessary. This produces a list of matching files at /tmp/file_list.

The problem is that this file might have duplicates in it. Hence, we can replace the duplicates with |sort|uniq. The sort part places the duplicates adjacent to each other, so that uniq can remove them. Then you can cat these files together using xargs (with each file name seperated by newline \n). Hence,

</tmp/file_list sort | uniq | xargs -d "\n" cat > final_file.txt

Unlike the other answers, this has two steps in it, and a temporary file, so I'd really only recommend it if you have multiple patterns to find.

Answer 4

Depending on your shell and environment, you could so something like this (in bash)

while IFS= read -r -d '' file; do
  if grep -qim1 'searchPattern1\|searchPattern2\|searchPattern3' "$file"; then
    cat "$file" >> some/other/file
  fi
done < <(find ./recup*/ -name '*.txt' -print0)

If you want to separate the results according to pattern, you could modify that to something like

while IFS= read -r -d '' file; do
  if grep -qim1 'searchPattern1' "$file"; then
    cat "$file" >> some/other/file1
  elif grep -qim1 'searchPattern2' "$file"; then
    cat "$file" >> some/other/file2
  elif grep -qim1 'searchPattern3' "$file"; then
    cat "$file" >> some/other/file3
  fi
done < <(find ./recup*/ -name '*.txt' -print0)