5

I'm using grep to search through 1 TB of files. I want to grep filenames and put the names in a text file AND I want to cp all files with a match to the dir /home/user/matches. I want to both tasks without searching through all my files with grep twice.

I had the idea to put filenames output into a text file with grep

grep -ril "xxx" . >> /home/user/matches/output-filename.txt

And now use output-filename.txt as input for cp and make cp execute line per line. How do I do that? awk? Or do you guys have other ideas to avoid searching through all files twice

2
  • Hard to tell what you are trying to do. However is looks like a job for xargs and tee. Commented Dec 19, 2022 at 22:25
  • What do you mean by "directions" in the title? All the copying is to the same directory, not bidirectional according to some specification. Maybe just just meant reading paths (filenames) to copy? Commented Dec 21, 2022 at 5:37

3 Answers 3

8

File paths are sequences of bytes other than 0; they're not necessarily text let alone lines of texts. In particular, a file path

  • may contain newline characters
  • may contain sequences of bytes that don't form valid characters
  • may be longer than LINE_MAX

The GNU implementation of grep (the one that added the -r option), can print the paths in a non-text format with -Z that is post-processable safely. For instance, GNU xargs can process that format with its -0 option:

xargs -r0 -a <(
  grep -rilZ xxx . |
   tee file.list
)  cp -it /home/user/matches --

(here also assuming GNU cp for its -t option)

If you want to print that list in a text¹ format that is understandable by a human, with GNU printf:

xargs -r0a file.list printf '%q\n'

¹ Well, it should ensure bytes that can't be decoded as characters are rendered as $'\234' representations. Same for control characters including newline, which is rendered as $'\n'. That addresses the first two points above, but it doesn't guarantee the output will have lines shorter than LINE_MAX (but then again, GNU implementations of standard text utilities generally don't have a limit on length of lines they support).

3
  • Is --reflink=auto the default for cp? If not, specifying it explicitly will save a lot of I/O if the files are on the same filesystem, and the FS supports reflinks (like BRTFS). (I'd normally have used ln for something like this, although that does disturb the ctime on each of the files being linked; that's an advantage for making reflinks.) Commented Dec 21, 2022 at 5:35
  • @PeterCordes see Why is cp --reflink=auto not the default behaviour?. Using reflink or not is rather orthogonal to this Q&A so I'll leave it out. Your comment is still useful though. Commented Dec 21, 2022 at 14:43
  • Ok, good, it finally is the default, many years after the option was introduced. (In coreutils 9.0, the first major release since that 2013 question). Thanks. Commented Dec 21, 2022 at 20:30
5

You can use find with a chained pair of exec commands. Perhaps not the most efficient solution as it invokes grep for every file (and cp for every matched file), but it will work regardless of the characters in the file name:

mkdir -p ~/matches
find -type f -exec grep -il 'xxx' {} \; -exec cp -p {} ~/matches/ \; > ~/matches/output-filename.txt
3

Like this using tee, xargs commands (using NULL byte \0 as end of filename separator) and GNU grep:

Handle filenames with spaces, newlines etc...

mkdir -p ~/matches
grep -Zril "xxx" . |
    tee ~/matches/output-filename.txt |
    xargs -0 -I {} cp {} ~/matches/
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .