using parallel to process unique input files to unique output files

Question

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to a unique file (aka, file_1.input needs to be captured in file_1.output, and so on).

Pre-parallel, I would just iterate over each file in the directory and perform my command, while doing some sort of timer/counting technique to not overwhelm the processors (assuming that each process had a constant runtime). However, I know that won't always be the case, so using a "parallel" like solution seems the best way to get shell script multi-threading without writing custom code.

While I have thought of some ways to whip up parallel to process each of these files (and allowing me to manage my cores efficiently), they all seem hacky. I have what I think is a pretty easy use case, so would prefer to keep it as clean as possible (and nothing in the parallel examples seem to jump out as being my problem.

Any help would be appreciated!

input directory example:

> ls -l input_files/
total 13355
location1.txt
location2.txt
location3.txt
location4.txt
location5.txt

Script:

> cat proces_script.sh
#!/bin/sh

customScript -c 33 -I -file [inputFile] -a -v 55 > [outputFile]

Update: After reading Ole's answer below, I was able to put together the missing pieces for my own parallel implementation. While his answer is great, here is my addition research and notes I took:

Instead of running my full process, I figured to start with a proof of concept command to prove out his solution in my environment. See my two different implementations (and notes):

find /home/me/input_files -type f -name *.txt | parallel cat /home/me/input_files/{} '>' /home/me/output_files/{.}.out

Uses find (not ls, that can cause issues) to find all applicable files within my input files directory, and then redirects their contents to a separate directory and file. My issue from above was reading and redirecting (the actual script was simple), so replacing the script with cat was a fine proof of concept.

parallel cat '>' /home/me/output_files/{.}.out :::  /home/me/input_files/*

This second solution uses parallel's input variable paradigm to read the files in, however for a novice, this was much more confusing. For me, using find a and pipe met my needs just fine.

Answer 1

GNU Parallel is designed for this kind of tasks:

parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input

or:

ls | parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output

It will run one jobs per CPU core.

You can install GNU Parallel simply by:

wget https://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 2

The standard way to do this is to setup a queue and spawn any number of workers that know how to pull something from the queue and process it. You can use a fifo (aka named pipe) for communication between these processes.

Below is a naive example to demonstrate the concept.

A simple queue script:

#!/bin/sh
mkfifo /tmp/location-queue
for i in inputfiles/*; do
  echo $i > /tmp/location-queue
done
rm /tmp/location-queue

And a worker:

#!/bin/sh
while read file < /tmp/location-queue; do
  process_file "$file"
done

process_file could be defined somewhere in your worker, and it can do whatever you need it to do.

Once you have those two pieces, you can have a simple monitor that starts up the queue process and any number of worker processes.

Monitor script:

#!/bin/sh
queue.sh &
num_workers="$1"
i=0
while [ $i < $num_workers ]; do
  worker.sh &
  echo $! >> /tmp/worker.pids
  i=$((i+1))
done
monitor_workers

There you have it. If you actually do this, it's better to setup the fifo in the monitor, and pass the path to both the queue and the workers, so they are not coupled and not stuck to a specific location for the fifo. I set it up this way in the answer specifically so it's clear that what you're using as you read it.

Answer 3

Another example:

ls *.txt | parallel 'sort {} > {.}.sorted.txt'

I found the other examples unnecessarily complex, when in most cases the above is what you may have been searching for.

Answer 4

A commonly available tool that can do parallelization is make. GNU make and a few others have a -j option to perform parallel builds.

.SUFFIXES: .input .output
.input.output:
        process_one_file <$< >[email protected]
        mv -f [email protected] $@

Run make like this (I assume your file names don't contain any special characters, make is no good with those):

make -j 4 $(for x in *.input; do echo ${x%.*}.output; done)

Answer 5

This is to perform a the same command on a large set of files in the current directory:

#!/bin/sh
trap 'worker=`expr $worker - 1`' USR1  # free up a worker
worker=0  # current worker
num_workers=10  # maximum number of workers
for file in *.txt; do
    if [ $worker -lt $num_workers ]; then
        {   customScript -c 33 -I -file $file -a -v 55 > `basename $file .txt`.outtxt 
            kill -USR1 $$ 2>/dev/null  # signal parent that we're free
        } &
        echo $worker/$num_worker $! $file  # feedback to caller
        worker=`expr $worker + 1`
    else
        wait # for a worker to finish
    fi
done

This runs the customScript on each txt file, putting the output in outtxt files. Change as you need. The key to getting this to work is the signal processing, using SIGUSR1 so the child process can let the parent process know that it is done. Using SIGCHLD won't work since most of the statements in the script will generate SIGCHLD signals to the shell script. I tried this replacing your command with sleep 1, the program used 0.28s of user cpu and 0.14s of system cpu; this was only on about 400 files.

Answer 6

Or simply use xargs -P , no need to install additonal software:

find . -type f -print0 | xargs -0 -I'XXX' -P4 -n1 custom_script -input "XXX" -output "XXX.out"

A bit of explanation for the options:

• -I'XXX' sets the string that will be replaced in the command template with file name
• -P4 will run 4 processes in parallel
• -n1 will put only one file per execution even though two XXX are found
• -print0 and -0 work together, letting you have special characters (like whitespace) in the file names