Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm learning a bit of Bash by using xargs to list whois records for a bunch of IP addresses. The command used is:

echo "$1" | tr "\n" "\0" | xargs -0 -n 1 -t -P 3 -I %  sh -c 'echo "\n 44rBegin whois record -- \n"; whois -h whois.arin.net % ; echo "\n 44rEnd whois record -- \n"'

The executed commands are:

sh -c echo "\n 44rBegin whois record -- \n"; whois -h whois.arin.net 206.190.36.45 ; echo "\n 44rEnd whois record -- \n"
sh -c echo "\n 44rBegin whois record -- \n"; whois -h whois.arin.net 212.146.69.237 ; echo "\n 44rEnd whois record -- \n"
sh -c echo "\n 44rBegin whois record -- \n"; whois -h whois.arin.net 77.238.184.24 ; echo "\n 44rEnd whois record -- \n"

I want the output to appear as if each block of commands executed using sh -c is executed sequentially. Instead my output is something like:

44rBegin whois record

44rBegin whois record

44rBegin whois record

whois1 output

44rEnd whois record --

whois2 output

44rEnd whois record --

whois3 output

44rEnd whois record --

How can I fix this problem?

share|improve this question

3 Answers 3

up vote 1 down vote accepted

Any time you have multiple processes outputting to the same terminal (or file) in parallel, you run the risk of their output getting interspersed (unless you arrange to do some sort of locking or use low-level system calls like write to files opened in append-only mode).

As a first step, you can minimize, but not totally eliminate, the problem by having each shell invocation use command substitution: run the whois command as a subprocess, capturing its output, then output everything combined into one printf operation.

xargs -0 -n 1 -P 3 -I %% sh -c 'printf "\n%s\n%s\n%s\n" " 44rBegin whois record -- " "$(whois -h whois.arin.net %%)" " 44rEnd whois record -- "'

Even better, if you have the flock program available, you can use it to lock each call to that combined printf:

xargs -0 -n 1 -P 3 -I %% sh -c 'who="$(whois -h whois.arin.net %%)"; flock /tmp/who.lock printf "\n%s\n%s\n%s\n" " 44rBegin whois record -- " "$who" " 44rEnd whois record -- "'
share|improve this answer
    
I was going to amend my answer to include this. This is a good idea, but from playing around the shell behaves really weird here. It'll output the first 4k bytes in a single write() operation, but after that it starts going line-by-line. Still, better than nothing at all though. –  Patrick Jul 11 at 18:25
    
@Patrick Thanks, I forgot how big those whois records can be. I added a solution that uses flock. –  Mark Plotnick Jul 11 at 22:49

In this specific case, you're passing -P 3 to xargs.

   -P max-procs
          Run up to max-procs processes at a time; the default is 1.  If max-procs  is
          0,  xargs  will run as many processes as possible at a time.  Use the -n op‐
          tion with -P; otherwise chances are that only one exec will be done.

Because you're running them in parallel, the're all going to write their output at the same time.

If you copied this command from somewhere else, I would recommend that you research to understand what it is you're copying. Doing otherwise can be dangerous.

share|improve this answer
    
the command is not copied. THe idea was that if I have n=100 P=3 ( or anumber close to that) would make the whole batch execution faster. At least from what i understand. And how can i make that whole block (echo; whois;echo) inseparable but the sh parallel? –  tubis Jul 11 at 16:19
    
It's called locking and it's really hard in sh. I find it much easier to spool to specific files and then cat them later. –  Joshua Jul 11 at 20:01

Yes, the output is mixed because of xargs -P. You're executing several subprocesses in parallel, and there is nothing to coordinate their output: they're all writing output whenever they want and it all gets mixed up.

Use GNU Parallel, which is a far more powerful tool to do the same job as xargs -P. Its default is to group output from each job together.

echo "$1" | parallel -t -P 3 sh -c 'echo "\n 44rBegin whois record -- \n"; whois -h whois.arin.net $0; echo "\n 44rEnd whois record -- \n"'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.