How to stop xargs from badly merging output from multiple processes?

Question

I'm using xargs with the option --max-args=0 (alternatively -P 0).

However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end up with lines such as:

<start-of-line-1><line-2><end-of-line-1>

As I'm using egrep with ^ in my pattern on the whole xargs output this is messing up my result.

Is there some way to force xargs to write the process outputs in order (any order, as long as the output of one process is contiguous)?

Or some other solution?

Edit: more details about the use case:

I want to download and parse web pages from different hosts. As every page takes about a second to load and there are a few dozen pages I want to parallelize the requests.

My command has the following form:

echo -n $IPs | xargs --max-args=1 -I {} --delimiter ' ' --max-procs=0 \
wget -q -O- http://{}/somepage.html | egrep --count '^string'

I use bash and not something like Perl because the host IPs (the $IPs variable) and some other data comes from an included bash file.

Answer 1

GNU Parallel is specifcally designed to solve this problem:

echo -n $IPs | parallel -d ' ' -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string'

If your IPs are in a file it is even prettier:

cat IPs | parallel -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string'

To learn more watch the intro video: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 2

This should do the trick:

echo -n $IPs | xargs --max-args=1 -I {} --delimiter ' ' --max-procs=0 \
  sh -c "wget -q -O- 'http://{}/somepage.html' | egrep --count '^string'" | \
  { NUM=0; while read i; do NUM=$(($NUM + $i)); done; echo $NUM; }

The idea here is to make separate counts and sum these at the end. Might fail if the separate counts are big enough to be mixed, but it should not be the case.