18

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to a unique file (aka, file_1.input needs to be captured in file_1.output, and so on).

Pre-parallel, I would just iterate over each file in the directory and perform my command, while doing some sort of timer/counting technique to not overwhelm the processors (assuming that each process had a constant runtime). However, I know that won't always be the case, so using a "parallel" like solution seems the best way to get shell script multi-threading without writing custom code.

While I have thought of some ways to whip up parallel to process each of these files (and allowing me to manage my cores efficiently), they all seem hacky. I have what I think is a pretty easy use case, so would prefer to keep it as clean as possible (and nothing in the parallel examples seem to jump out as being my problem.

Any help would be appreciated!

input directory example:

> ls -l input_files/
total 13355
location1.txt
location2.txt
location3.txt
location4.txt
location5.txt

Script:

> cat proces_script.sh
#!/bin/sh

customScript -c 33 -I -file [inputFile] -a -v 55 > [outputFile]

Update: After reading Ole's answer below, I was able to put together the missing pieces for my own parallel implementation. While his answer is great, here is my addition research and notes I took:

Instead of running my full process, I figured to start with a proof of concept command to prove out his solution in my environment. See my two different implementations (and notes):

find /home/me/input_files -type f -name *.txt | parallel cat /home/me/input_files/{} '>' /home/me/output_files/{.}.out

Uses find (not ls, that can cause issues) to find all applicable files within my input files directory, and then redirects their contents to a separate directory and file. My issue from above was reading and redirecting (the actual script was simple), so replacing the script with cat was a fine proof of concept.

parallel cat '>' /home/me/output_files/{.}.out :::  /home/me/input_files/*

This second solution uses parallel's input variable paradigm to read the files in, however for a novice, this was much more confusing. For me, using find a and pipe met my needs just fine.

6 Answers 6

28

GNU Parallel is designed for this kind of tasks:

parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input

or:

ls | parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output

It will run one jobs per CPU core.

You can install GNU Parallel simply by:

wget https://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

1
  • Great answer (and major points for reading my request of using parallel).
    – J Jones
    Feb 28, 2012 at 3:17
5

The standard way to do this is to setup a queue and spawn any number of workers that know how to pull something from the queue and process it. You can use a fifo (aka named pipe) for communication between these processes.

Below is a naive example to demonstrate the concept.

A simple queue script:

#!/bin/sh
mkfifo /tmp/location-queue
for i in inputfiles/*; do
  echo $i > /tmp/location-queue
done
rm /tmp/location-queue

And a worker:

#!/bin/sh
while read file < /tmp/location-queue; do
  process_file "$file"
done

process_file could be defined somewhere in your worker, and it can do whatever you need it to do.

Once you have those two pieces, you can have a simple monitor that starts up the queue process and any number of worker processes.

Monitor script:

#!/bin/sh
queue.sh &
num_workers="$1"
i=0
while [ $i < $num_workers ]; do
  worker.sh &
  echo $! >> /tmp/worker.pids
  i=$((i+1))
done
monitor_workers

There you have it. If you actually do this, it's better to setup the fifo in the monitor, and pass the path to both the queue and the workers, so they are not coupled and not stuck to a specific location for the fifo. I set it up this way in the answer specifically so it's clear that what you're using as you read it.

4
  • How is the monitor smart enough to pause spawning on new workers until the next has finished (aka, where does $i ever get decremented)? ---- Answering my own edit, the workers never go away, they just process files until all processing has been exhausted (hence the while loop within the 'processors' as well).
    – J Jones
    Feb 20, 2012 at 13:46
  • What's the "monitor_workers" line at the end of the monitor script performing?
    – J Jones
    Feb 20, 2012 at 13:51
  • @JJones - monitor_workers is just like process_file - it's a function that does whatever you want. About the monitor - you were right; it should save the pids of its workers (so it can send a kill signal) and the counter needs to be incremented when it starts a worker. I've edited the answer to include that. Feb 20, 2012 at 14:12
  • I really appreciate your work, but I think you should use GNU's parallel. I think it is your idea, fully implemented.
    – motobói
    Sep 24, 2015 at 1:28
5

Another example:

ls *.txt | parallel 'sort {} > {.}.sorted.txt'

I found the other examples unnecessarily complex, when in most cases the above is what you may have been searching for.

4

A commonly available tool that can do parallelization is make. GNU make and a few others have a -j option to perform parallel builds.

.SUFFIXES: .input .output
.input.output:
        process_one_file <$< >[email protected]
        mv -f [email protected] $@

Run make like this (I assume your file names don't contain any special characters, make is no good with those):

make -j 4 $(for x in *.input; do echo ${x%.*}.output; done)
1
  • imho this is the most clever solution :)
    – h4unt3r
    Jun 4, 2014 at 18:14
3

This is to perform a the same command on a large set of files in the current directory:

#!/bin/sh
trap 'worker=`expr $worker - 1`' USR1  # free up a worker
worker=0  # current worker
num_workers=10  # maximum number of workers
for file in *.txt; do
    if [ $worker -lt $num_workers ]; then
        {   customScript -c 33 -I -file $file -a -v 55 > `basename $file .txt`.outtxt 
            kill -USR1 $$ 2>/dev/null  # signal parent that we're free
        } &
        echo $worker/$num_worker $! $file  # feedback to caller
        worker=`expr $worker + 1`
    else
        wait # for a worker to finish
    fi
done

This runs the customScript on each txt file, putting the output in outtxt files. Change as you need. The key to getting this to work is the signal processing, using SIGUSR1 so the child process can let the parent process know that it is done. Using SIGCHLD won't work since most of the statements in the script will generate SIGCHLD signals to the shell script. I tried this replacing your command with sleep 1, the program used 0.28s of user cpu and 0.14s of system cpu; this was only on about 400 files.

2
  • How is the 'wait' smart enough to take the same file that is currently being iterated over and re-enter the sibling "if" statement?
    – J Jones
    Feb 21, 2012 at 13:09
  • It is not the wait that is 'smart' enough; but it will return after getting the SIGUSR1 signal. The child/worker sends a SIGUSR1 to the parent, which is caught (trap), and decrements $worker (trap clause) and return abnormally from wait, allowing the if [ $worker -lt $num_workers ] clause to execute.
    – Arcege
    Feb 21, 2012 at 13:58
0

Or simply use xargs -P , no need to install additonal software:

find . -type f -print0 | xargs -0 -I'XXX' -P4 -n1 custom_script -input "XXX" -output "XXX.out"

A bit of explanation for the options:

  • -I'XXX' sets the string that will be replaced in the command template with file name
  • -P4 will run 4 processes in parallel
  • -n1 will put only one file per execution even though two XXX are found
  • -print0 and -0 work together, letting you have special characters (like whitespace) in the file names

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .