How to use -I{} with xargs' with an -n greater than 1?

Question

I have a very large number of files (tens of thousands) that I need to grep through for a specific string. A small handful of the files have spaces in them. There are so many files the process creation overhead -n1 is actually bigger than the file search itself.

This works, but is unusably slow:

cat filelist | xargs -I{} grep mystring '{}'

So, I want to pass 1000 arguments to each grep instance, like this:

cat filelist | xargs -n1000 -I{} grep mystring '{}'

But this doesn't work. It seems the {} only works when -n = 1?!

Example:

Too many processes, correct output:
$ seq 1 10 | xargs -I{} -n1 echo "<{}>"
<1>
<2>
<3>
<4>
<5>
<6>
<7>
<8>
<9>
<10>

Good number of processes, and... what?
$ seq 1 10 | xargs -I{} -n2 echo "<{}>"
<{}> 1 2
<{}> 3 4
<{}> 5 6
<{}> 7 8
<{}> 9 10

Maybe I can use find instead.

Answer 1

Yes, -I only works with for one argument at a time. With -I, the input is also parsed into arguments in a different way than without (using -n or not).

With -I{} you get a word for each non-empty line (except it's still possible to embed a newline by quoting it with backslash), with the leading, but not trailing blank characters (the list of which varies with the implementation and locale for some) removed. Quotes (", ' and \ are still processed, in a different way from sh's though).

Without -I{}, words are whitespace (at least SPC, TAB and NL) delimited, and quotes processed.

Compare:

$ printf ' a "b c" \n' | xargs -n1 printf '<%s>\n'
<a>
<b c>
$ printf ' a "b c" \n' | xargs -I{} printf '<%s>\n' {}
<a b c >

IMO, xargs is a bit of a mess, the only reliable/useful ways to use it are with the -0 and -d GNU extensions.

If you want to run a command with more than one argument at a time and use a different place-holder for each, best is to use sh:

xargs < filelist -r -n2 sh -c 'printf "1: %s\n2: %s\n" "$1" "$2"' sh

Here, xargs passes 2 arguments at a time to sh, and sh does the place holding with "$1" and "$2" (see also "$@" to pass all arguments at once).

That's with the default word tokenising of xargs. If filelist is meant to contain one file per line, you'd use GNU xargs's -d '\n'.

For your grep example, you don't need -n nor -I though, just:

xargs < filelist grep mystring

Then xargs will pass as many arguments as possible to grep (the arguments are added at the end). We can do without -r here (a GNU extension) as if filelist is all blank, still running grep without file arguments (which -r prevents) should be harmless as it would search for mystring at the end of filelist.

You may however want to use the -H option of GNU grep, or run it as:

xargs < filelist grep mystring /dev/null

to make sure that grep always prints the file name when it finds a match even if filelist contains only one word.

Answer 2

GNU Parallel has fixed this issue. -X will include the context:

$ seq 1 10 | parallel -j1 -qX echo "<{}>"
<1> <2> <3> <4> <5> <6> <7> <8> <9> <10>

-m will not:

$ seq 1 10 | parallel -j1 -qm echo "<{}>"
<1 2 3 4 5 6 7 8 9 10>