I have written a bash script which is in following format:

start=$(date +%s)

rm -f $inFile $outFile

while read line

    -- Block of Commands

done < "$inFile"

end=$(date +%s)


echo "Program has finished execution in $runtime seconds."

The while loop will read from $inFile, perform some activity on the line and dump the result in $outFile.

As the $inFile is 3500+ lines long, the script would take 6-7 hours for executing completely. In order to minimize this time, I am planning to use multi-threading or forking in this script. If I create 8 child processes, 8 lines from the $inFile would be processed simultaneously.

How can this be done?

  • Be careful: different scripts will need to write to different outfiles. Also your script as written deletes the input file as the first action! – pjc50 Aug 1 '14 at 10:59

GNU parallel is made for just this sort of thing. You can run your script many times at once, with different data from your input piped in for each one:

cat input.txt | parallel --pipe your-script.sh

By default it will spawn processes according to the number of processors on your system, but you can customise that with -j N.

A particularly neat trick is the shebang-wrapping feature. If you change the first line of your Bash script to:

#!/usr/bin/parallel --shebang-wrap --pipe /bin/bash

and feed it data on standard input then it will all happen automatically. This is less useful when you have cleanup code that has to run at the end, which you may do.

There are a couple of things to note. One is that it will chop up your input into sequential chunks and use those one at a time - it doesn't interleave lines. The other is that it those chunks are split by size, without regard for how many records there are. You can use --block N to set a different block size in bytes. In your case, no more than an eighth of the file size should be about right. Your file sounds like it might be small enough to end up all in a single block otherwise, which would defeat the purpose.

There are a lot of options for particular different use cases, but the tutorial covers things pretty well. Options you might also be interested in include --round-robin and --group.

| improve this answer | |
  • 1
    Did you test that shebang line? Shebangs with multiple arguments are unportable. On Linux, #!a b c will result in ["b c"], while on some other systems, it will result in ["b", "c"]. – nyuszika7h Aug 1 '14 at 14:33
  • 1
    It reparses its own arguments when used in this way (otherwise the option wouldn't be much use). – Michael Homer Aug 1 '14 at 18:58
  • @MichaelHomer I need to use GNU parallel for scraping of HTML pages. Could you please go through this thread unix.stackexchange.com/questions/277609/… – Swatesh Pakhare Apr 19 '16 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.