Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to a unique file (aka, file_1.input needs to be captured in file_1.output, and so on).

Pre-parallel, I would just iterate over each file in the directory and perform my command, while doing some sort of timer/counting technique to not overwhelm the processors (assuming that each process had a constant runtime). However, I know that won't always be the case, so using a "parallel" like solution seems the best way to get shell script multi-threading without writing custom code.

While I have thought of some ways to whip up parallel to process each of these files (and allowing me to manage my cores efficiently), they all seem hacky. I have what I think is a pretty easy use case, so would prefer to keep it as clean as possible (and nothing in the parallel examples seem to jump out as being my problem.

Any help would be appreciated!

input directory example:

> ls -l input_files/
total 13355


> cat proces_script.sh

customScript -c 33 -I -file [inputFile] -a -v 55 > [outputFile]

Update: After reading Ole's answer below, I was able to put together the missing pieces for my own parallel implementation. While his answer is great, here is my addition research and notes I took:

Instead of running my full process, I figured to start with a proof of concept command to prove out his solution in my environment. See my two different implementations (and notes):

find /home/me/input_files -type f -name *.txt | parallel cat /home/me/input_files/{} '>' /home/me/output_files/{.}.out

Uses find (not ls, that can cause issues) to find all applicable files within my input files directory, and then redirects their contents to a separate directory and file. My issue from above was reading and redirecting (the actual script was simple), so replacing the script with cat was a fine proof of concept.

parallel cat '>' /home/me/output_files/{.}.out :::  /home/me/input_files/*

This second solution uses parallel's input variable paradigm to read the files in, however for a novice, this was much more confusing. For me, using find a and pipe met my needs just fine.

share|improve this question
up vote 10 down vote accepted

GNU Parallel is designed for this kind of tasks:

parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input


ls | parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output

It will run one jobs per CPU core.

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
Great answer (and major points for reading my request of using parallel). – J Jones Feb 28 '12 at 3:17

The standard way to do this is to setup a queue and spawn any number of workers that know how to pull something from the queue and process it. You can use a fifo (aka named pipe) for communication between these processes.

Below is a naive example to demonstrate the concept.

A simple queue script:

mkfifo /tmp/location-queue
for i in inputfiles/*; do
  echo $i > /tmp/location-queue
rm /tmp/location-queue

And a worker:

while read file < /tmp/location-queue; do
  process_file "$file"

process_file could be defined somewhere in your worker, and it can do whatever you need it to do.

Once you have those two pieces, you can have a simple monitor that starts up the queue process and any number of worker processes.

Monitor script:

queue.sh &
while [ $i < $num_workers ]; do
  worker.sh &
  echo $! >> /tmp/worker.pids

There you have it. If you actually do this, it's better to setup the fifo in the monitor, and pass the path to both the queue and the workers, so they are not coupled and not stuck to a specific location for the fifo. I set it up this way in the answer specifically so it's clear that what you're using as you read it.

share|improve this answer
How is the monitor smart enough to pause spawning on new workers until the next has finished (aka, where does $i ever get decremented)? ---- Answering my own edit, the workers never go away, they just process files until all processing has been exhausted (hence the while loop within the 'processors' as well). – J Jones Feb 20 '12 at 13:46
What's the "monitor_workers" line at the end of the monitor script performing? – J Jones Feb 20 '12 at 13:51
@JJones - monitor_workers is just like process_file - it's a function that does whatever you want. About the monitor - you were right; it should save the pids of its workers (so it can send a kill signal) and the counter needs to be incremented when it starts a worker. I've edited the answer to include that. – Shawn J. Goff Feb 20 '12 at 14:12
I really appreciate your work, but I think you should use GNU's parallel. I think it is your idea, fully implemented. – motobói Sep 24 '15 at 1:28

A commonly available tool that can do parallelization is make. GNU make and a few others have a -j option to perform parallel builds.

.SUFFIXES: .input .output
        process_one_file <$< >$@.tmp
        mv -f $@.tmp $@

Run make like this (I assume your file names don't contain any special characters, make is no good with those):

make -j 4 $(for x in *.input; do echo ${x%.*}.output; done)
share|improve this answer
imho this is the most clever solution :) – h4unt3r Jun 4 '14 at 18:14

This is to perform a the same command on a large set of files in the current directory:

trap 'worker=`expr $worker - 1`' USR1  # free up a worker
worker=0  # current worker
num_workers=10  # maximum number of workers
for file in *.txt; do
    if [ $worker -lt $num_workers ]; then
        {   customScript -c 33 -I -file $file -a -v 55 > `basename $file .txt`.outtxt 
            kill -USR1 $$ 2>/dev/null  # signal parent that we're free
        } &
        echo $worker/$num_worker $! $file  # feedback to caller
        worker=`expr $worker + 1`
        wait # for a worker to finish

This runs the customScript on each txt file, putting the output in outtxt files. Change as you need. The key to getting this to work is the signal processing, using SIGUSR1 so the child process can let the parent process know that it is done. Using SIGCHLD won't work since most of the statements in the script will generate SIGCHLD signals to the shell script. I tried this replacing your command with sleep 1, the program used 0.28s of user cpu and 0.14s of system cpu; this was only on about 400 files.

share|improve this answer
How is the 'wait' smart enough to take the same file that is currently being iterated over and re-enter the sibling "if" statement? – J Jones Feb 21 '12 at 13:09
It is not the wait that is 'smart' enough; but it will return after getting the SIGUSR1 signal. The child/worker sends a SIGUSR1 to the parent, which is caught (trap), and decrements $worker (trap clause) and return abnormally from wait, allowing the if [ $worker -lt $num_workers ] clause to execute. – Arcege Feb 21 '12 at 13:58

Another example:

ls *.txt | parallel 'sort {} > {.}.sorted.txt'

I found the other examples unnecessarily complex, when in most cases the above is what you may have been searching for.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.