7

First, I think the fact that I'm using cygwin is highly relevant here.

In theory, I already know how to do this:

cat file | xargs grep pattern

The problem is, some of the file paths in file have spaces. The file looks like this:

subdir/foo/bar.html
subdir/f o o/ba r.html
subdir/~foo/bar.html

This causes errors. I read how to solve this: use xargs -0. But I don't know how to make cat output null terminated line endings, so i think that means it squashes the whole file into one line. As a result, it gives this error:

xargs: argument line too long

Update: it turned out the file I was reading from had some paths that no longer exist. Coincidentally, all of these paths had a ~ in them, so I mistakenly thought that that was an issue. Everything about the spaces still stands though. Turns out those files simply didn't exist.

4
  • @cas the file doesn't contain patterns, the file contains paths to files i want to search. It's the output of a find – Daniel Kaplan Jul 19 at 7:42
  • 1
    ah, okay. is it one line per filename? if so, use xargs -d '\n' grep pattern < file. better yet, since it's the output of find, use find ... -print0 | xargs -0 grep pattern – cas Jul 19 at 7:44
  • @DanielKaplan You mentioned having problems with ~. Is the problem that xargs is trying to expand them (even when you don't want it to)? – Steve Summit Jul 19 at 16:51
  • ~ has no special meaning in the middle of a path. At the beginning, ~/ means the current user's home directory, and ~username/ means that user's home directory. What do you intend with subdir/~foo? – Barmar Jul 21 at 20:35
13

Use xargs -d '\n' if the input file contains one filename per line. e.g.

xargs -d '\n' grep pattern < file

If your filenames start with ~ to indicate "my home directory", you first need to replace the ~ symbols with your actual home directory. For example:

sed -e "s=^~=$HOME=" file | xargs -d '\n' grep pattern

Note that this sed script is double-quoted because we want to interpolate the variable $HOME into the sed script, and uses = as the delimiter for the s operator because $HOME is going to contain /s (but is unlikely to contain an =).

Or, if you're using find, use find's -print0 option combined with xargs' -0 option. e.g.

find ... -print0 | xargs -0 grep pattern

or just use find's -exec option:

find ... -exec grep pattern {} +
9
  • Only that first one is useful to me as I can no longer use find to get the same file list. But when I tried it, it seemed to have trouble with filenames that have ~ in it. – Daniel Kaplan Jul 19 at 7:53
  • @Kusalananda that gave me this error xargs: -: No such file or directory – Daniel Kaplan Jul 19 at 7:53
  • @DanielKaplan You'll always have issues with pathnames containing ~ as this is something that the shell needs to expand. You may need to insert an explicit substitution from ~ to whatever your home directory path is. – Kusalananda Jul 19 at 7:55
  • 1
    BTW, it's unusual for the output of find to contain ~ characters. If I run find ~/ on my system, I get output lines starting with /home/cas/, not with ~/. – cas Jul 19 at 8:05
  • 1
    @cas the paths themselves have a ~ in them. e.g., ./foo/~bar/baz.html – Daniel Kaplan Jul 19 at 15:45
3

Another answer mentioned xargs -d '\n', which is the preferred solution. (In my opinion it's so preferred that it's what xargs should do by default, but that's another story.)

But not all versions of xargs support -d. If you're stuck with one that doesn't, you can try

tr '\n' '\0' < file | xargs -0

(Unfortunately not all versions of tr can substitute null characters like this, so it's still not a universal solution.)

1

Remember: xargs will put the whole content of file as command line arguments to grep ... The error you met is probably due to too much filename in your file resulting in a commmand line too long.

If you want to search for files which content match pattern, i propose the following iterative method:

gcolangelo@debian-test:/tmp$ cat test_files 
/dir1/file1 with space
/dir2/file2 with space
/dir3 with space/file3

gcolangelo@debian-test:/tmp$ while read filepath ; do echo "-- coin ${filepath} --" ; done < test_files 
-- coin /dir1/file1 with space --
-- coin /dir2/file2 with space --
-- coin /dir3 with space/file3 --
-- coin  --
2
  • 2
    The first sentence is true if you add "when using nul as the delimiter, since the file does not contain any nuls". – Kusalananda Jul 19 at 7:50
  • xargs should never create a command line too long. It will split up the input into batches that fit the allowed length. – Barmar Jul 21 at 20:37
0

In some contexts it might be beneficial to use a for loop instead of xargs (e.g. when you want to do execute more than one command):

while read file
do
    grep pattern "$file"
done < file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.