1

How to execute a bash script in parallel for each line ? Actually, I will be tailing to log file and for each line found, I want to execute script in background something like below:

tailf logfile.log | grep 'patternline' | while read line ; do 
    bash scriptname.sh "$line" & ; 
done

I would like to know, how to perform above using xargs (OR any other suitable method) parallely and also how to limit processes..

Thanks in advance.

1

You'd like to read the xargs manual and look up the -L and the -P flags in there.

tail -f logfile.log | grep 'patternline' |
xargs -P 4 -L 1 bash scriptname.sh

This will execute at most four instances of the command at a time (-P 4), and with one line of input for each invocation (-L 1).

Add -t to xargs to see what gets executed.

  • output after after grep command will be a line of Json format, how to enclose input for xargs in double quotes.. ?? – AVJ Jun 21 '16 at 11:32
  • @AVJ Why? Can't you do that in the script? Also, do you know about jq? stedolan.github.io/jq – Kusalananda Jun 21 '16 at 11:37
  • @AVJ the double quoting is not needed here, is it? Just use "$*" in your script to get the complete line without word splitting. – Kusalananda Jun 21 '16 at 11:46
  • @AVJ If you really want to have double-quoted input, that must happen before xargs. Insert sed -e 's/^/"/' -e 's/$/"/' in the pipe before xargs. – Kusalananda Jun 21 '16 at 12:02
  • 1
    it worked using sed before xargs as u said... to parse json using jq, input to script should be enclosed in single or double quotes. – AVJ Jun 21 '16 at 13:35
0

GNU Parallel will do the quoting of the argument correctly for you:

tail -f logfile.log | grep 'patternline' | parallel bash scriptname.sh

On top of that it will default to 1 process per CPU core and it will make sure the output of two parallel jobs will not be mixed.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.