2

I have a very large number of files (tens of thousands) that I need to grep through for a specific string. A small handful of the files have spaces in them. There are so many files the process creation overhead -n1 is actually bigger than the file search itself.

This works, but is unusably slow:

cat filelist | xargs -I{} grep mystring '{}'

So, I want to pass 1000 arguments to each grep instance, like this:

cat filelist | xargs -n1000 -I{} grep mystring '{}'

But this doesn't work. It seems the {} only works when -n = 1?!

Example:

Too many processes, correct output:
$ seq 1 10 | xargs -I{} -n1 echo "<{}>"
<1>
<2>
<3>
<4>
<5>
<6>
<7>
<8>
<9>
<10>
Good number of processes, and... what?
$ seq 1 10 | xargs -I{} -n2 echo "<{}>"
<{}> 1 2
<{}> 3 4
<{}> 5 6
<{}> 7 8
<{}> 9 10

Maybe I can use find instead.

  • You can also think about grep with the recursive option. That's just one process then. – Tomasz May 10 at 17:20
5

Yes, -I only works with for one argument at a time. With -I, the input is also parsed into arguments in a different way than without (using -n or not).

With -I{} you get a word for each non-empty line (except it's still possible to embed a newline by quoting it with backslash), with the leading, but not trailing blank characters (the list of which varies with the implementation and locale for some) removed. Quotes (", ' and \ are still processed, in a different way from sh's though).

Without -I{}, words are whitespace (at least SPC, TAB and NL) delimited, and quotes processed.

Compare:

$ printf ' a "b c" \n' | xargs -n1 printf '<%s>\n'
<a>
<b c>
$ printf ' a "b c" \n' | xargs -I{} printf '<%s>\n' {}
<a b c >

IMO, xargs is a bit of a mess, the only reliable/useful ways to use it are with the -0 and -d GNU extensions.

If you want to run a command with more than one argument at a time and use a different place-holder for each, best is to use sh:

xargs < filelist -r -n2 sh -c 'printf "1: %s\n2: %s\n" "$1" "$2"' sh

Here, xargs passes 2 arguments at a time to sh, and sh does the place holding with "$1" and "$2" (see also "$@" to pass all arguments at once).

That's with the default word tokenising of xargs. If filelist is meant to contain one file per line, you'd use GNU xargs's -d '\n'.

For your grep example, you don't need -n nor -I though, just:

xargs < filelist grep mystring

Then xargs will pass as many arguments as possible to grep (the arguments are added at the end). We can do without -r here (a GNU extension) as if filelist is all blank, still running grep without file arguments (which -r prevents) should be harmless as it would search for mystring at the end of filelist.

You may however want to use the -H option of GNU grep, or run it as:

xargs < filelist grep mystring /dev/null

to make sure that grep always prints the file name when it finds a match even if filelist contains only one word.

0

GNU Parallel has fixed this issue. -X will include the context:

$ seq 1 10 | parallel -j1 -qX echo "<{}>"
<1> <2> <3> <4> <5> <6> <7> <8> <9> <10>

-m will not:

$ seq 1 10 | parallel -j1 -qm echo "<{}>"
<1 2 3 4 5 6 7 8 9 10>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.