Sending a list (text file) of files and pathnames to xargs

Question

I have a file that contains the paths of various files.

files.txt :

/path/to/file1.txt
/path/to/file2.txt
...
/path/to/fileN.txt

I want to use it in a script to process the files.

cat files.txt | xargs -P 12 -L 1 -I % /software/Linux/itt/idl/idl81/bin/idl -e program -args %

I basically want to send these paths as a string to this IDL program via xargs as an argument.

However when I use cat, it actually cats file1.txt, file2.txt, and outputs their content directly as an argument into this IDL program.

How can I get it to just send each and every line from this files.txt as a string to this IDL script?

Answer 1

You could use a small script with a loop like this

Let it be runIDL.sh

#!/bin/ksh
while (read filepath)
do    
    /software/Linux/itt/idl/idl81/bin/idl -e program -args "$filepath"
done < "$1"

It's simple, elegant and should work like a charm. Just run ./runIDL files.txt and that's it!

Answer 2

GNU Parallel may work for you:

cat files.txt | parallel /software/Linux/itt/idl/idl81/bin/idl -e program -args {}

It will default to 1 process per CPU core and it will make sure the output of two parallel jobs will not be mixed.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel