10

I managed to shoot myself where it hurts (really bad) by reformatting a partition that held valuable data. Of course it was not intentional, but it happened.

However, I managed to use testdisk and photorec to recover most of the data. So now I have all that data distributed over almost 25,000 directories. Most of the files are .txt files, while the rest are image files. There are more than 300 .txt files in each directory.

I can grep or use find to extract certain strings from the .txt files and output them to a file. For example, here's a line that I've used to verify that my data is in the recovered files:

find ./recup*/ -name '*.txt' -print | xargs grep -i "searchPattern"

I can output "searchPattern" to a file, but that just gives me that pattern. Here's what I really would like to accomplish:

Go through all the files and look for a specific string. If that string is found in a file, cat ALL the contents of that file to an output file. If the pattern is found in more than one file, append the contents of subsequent files to that output file. Note that I just don't want to output the pattern I'm searching for, but ALL the contents of the file in which the patterns is found.

I think this is doable, but I just don't know how to grab all the contents of a file after grepping a specific pattern from it.

  • So with the command you provided, it gives you the results you are looking for but you are looking to redirect the output to a text file? – ryekayo Sep 12 '14 at 0:17
  • After reading my question, that paragraph that starts with "Go through..." sounds just like psuedocode. Maybe I can get it code with a few lines of for/if Python code. Will give it a shot while I await a more informed response – Ami Sep 12 '14 at 0:19
  • It certainly is psuedocode, and Im sure you can find a way to do it in bash as well. – ryekayo Sep 12 '14 at 0:20
  • @ryekayo, Yes, it gives me the output, but that's just to find what file a specific type of data is in, which tells me that more of that data is in that file. So I want to grab everything in that file and write them to another file. – Ami Sep 12 '14 at 0:22
  • You can probably wrap that command in some kind of if statement or even a switch-case that can call a function that can cat out the contents based on the case or results of the if statement – ryekayo Sep 12 '14 at 0:22
9

If I understand your goal correctly, the following will do what you want:

find ./recup*/ -name '*.txt' -exec grep -qi "searchPattern" {} \; -exec cat {} \; > outputfile.txt

This will look for all *.txt files in ./recup*/, test each one for searchPattern, if it matches it'll cat the file. The output of all cated files will be directed into outputfile.txt.

Repeat for each pattern and output file.


If you have a very large number of directories matching ./recup*, you might end up with a argument list too long error. The simple way around this is to do something like this instead:

find ./ -mindepth 2 -path './recup*.txt' -exec grep -qi "searchPattern" {} \; -exec cat {} \; > outputfile.txt

This will match the full path. So ./recup01234/foo/bar.txt will be matched. The -mindepth 2 is so that it won't match ./recup.txt, or ./recup0.txt.

  • Yes, I think that will do it. And it gives me a base to work from. Since I'm going to be searching for multiple strings, I think a for/if bit of code, with multiple elif's will help me automate the task. Thank you – Ami Sep 12 '14 at 0:36
  • Thats even better than what I was thinking lol – ryekayo Sep 12 '14 at 0:41
  • That didn't seem to work. Got this error: "unable to execute /usr/bin/find: Argument list too long" – Ami Sep 12 '14 at 0:52
  • @Ami updated answer to provide a solution to that issue. – Patrick Sep 12 '14 at 1:19
  • 2
    @Ami If you are using multiple strings, it might be simpler to just save all positive file names to another file (grep -l), then |sort|uniq and cat from the file list. – Sparhawk Sep 12 '14 at 1:27
3

Rather than outputting your pattern, output the filename using "-l" on grep, and then use that as input to cat.

find ./recup*/ -name '*.txt' -print | xargs grep -li "searchPattern" | xargs cat

or

cat $( find ./recup*/ -name '*.txt' -print | xargs grep -li "searchPattern")

I suspect that you can fill in the remaining details. BTW, if you may have spaces or other odd characters in the filenames (unlikely in this specific case, but for future purposes), use -print0 on the find and -Z on the grep, combined with the -0 option on xargs to use null bytes between filenames rather than newlines.

find ./recup*/ -name '*.txt' -print0 | xargs -0 grep -Zli "searchPattern" | xargs -0 cat
  • 2
    I also like Patrick's "two -exec" option, except that it will cause a new fork (well, clone()) and exec for every file. Normally you can use \+ rather than \; to avoid that problem, but I don't know how that works with a pair of -exec args (I suspect "poorly"). Using a pair of xargs, you're only going to have a couple of new processes spawned, which should be faster with a lot of files. – dannysauer Sep 12 '14 at 2:41
  • This looks good, too. Thanks. One noob question: The cat after the last xargs should be outputting to a file, right? – Ami Sep 12 '14 at 2:47
  • When I first read it, I didn't think the question specified where the contents of the file should go. All three of these commands put the file(s) contents on STDOUT, so you'd just append (to the very end) >afile or |acommand or whatever is appropriate for your situation. :) – dannysauer Sep 12 '14 at 3:15
  • Good answer, I needed to cat pg_hba.conf sudo find /* -name pg_hba.conf | xargs sudo cat – App Work Jan 27 '17 at 0:06
  • This is a little off-topic, but I prefer using sudo xargs instead of xargs sudo. When you run xargs sudo, it builds the command line assuming the command is sudo cat args. But cat is in /bin, so then sudo runs /bin/cat args. If your command is in a longer directory, like /usr/local/bin, then the command sudo actually runs might result in a too-long command line and an error which is hard to track down. On top of that, sudo xargs just logs that you ran xargs, while xargs sudo logs the command with all of the arguments - resulting in some long sudo log lines. :) – dannysauer Jan 27 '17 at 19:22
1

This is not exactly optimal code, but it's very straightforward and will work fine if efficiency isn't an issue. The problem is that it'll grep through the files multiple times, even if the string has already been found in them.

Firstly, search for your strings and write the matching files to a list.

find ./recup*/ -name '*.txt' -execdir grep -il "searchPattern" {} >> /tmp/file_list \;

Repeat this step replacing searchPattern as necessary. This produces a list of matching files at /tmp/file_list.

The problem is that this file might have duplicates in it. Hence, we can replace the duplicates with |sort|uniq. The sort part places the duplicates adjacent to each other, so that uniq can remove them. Then you can cat these files together using xargs (with each file name seperated by newline \n). Hence,

</tmp/file_list sort | uniq | xargs -d "\n" cat > final_file.txt

Unlike the other answers, this has two steps in it, and a temporary file, so I'd really only recommend it if you have multiple patterns to find.

0

Depending on your shell and environment, you could so something like this (in bash)

while IFS= read -r -d '' file; do
  if grep -qim1 'searchPattern1\|searchPattern2\|searchPattern3' "$file"; then
    cat "$file" >> some/other/file
  fi
done < <(find ./recup*/ -name '*.txt' -print0)

If you want to separate the results according to pattern, you could modify that to something like

while IFS= read -r -d '' file; do
  if grep -qim1 'searchPattern1' "$file"; then
    cat "$file" >> some/other/file1
  elif grep -qim1 'searchPattern2' "$file"; then
    cat "$file" >> some/other/file2
  elif grep -qim1 'searchPattern3' "$file"; then
    cat "$file" >> some/other/file3
  fi
done < <(find ./recup*/ -name '*.txt' -print0)
  • What does the bit after "done" do? What I'll actually like is to modify that if block so that files which contain a matched pattern are written to a different. – Ami Sep 12 '14 at 2:01
  • It just lists the '.txt' files that are found, each being terminated by the null character (so that it's safe for filenames containing spaces and other characters). The while loop then reads that lists and does the grep / conditional cat part. – steeldriver Sep 12 '14 at 2:04
  • When I try to run the code, I get this error: ./recoverData.sh: Syntax error: "(" unexpected. That is coming from the brackets around the find command – Ami Sep 12 '14 at 3:12
  • What shell are you using? the process substitution syntax is specific to bash - hence my qualification "Depending on your shell and environment" – steeldriver Sep 12 '14 at 3:15
  • 1
    You can either execute the command(s) directly in an interactive bash shell, or put them in a file whose first line contains the shebang #!/bin/bash, make it executable with chmod +x recoverData.sh, and execute it using ./recoverData.sh. Do not use sh recoverData.sh since /bin/sh is likely a dash shell. – steeldriver Sep 12 '14 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.