2

I have a file that contains the paths of various files.

files.txt :

/path/to/file1.txt
/path/to/file2.txt
...
/path/to/fileN.txt

I want to use it in a script to process the files.

cat files.txt | xargs -P 12 -L 1 -I % /software/Linux/itt/idl/idl81/bin/idl -e program -args %

I basically want to send these paths as a string to this IDL program via xargs as an argument.

However when I use cat, it actually cats file1.txt, file2.txt, and outputs their content directly as an argument into this IDL program.

How can I get it to just send each and every line from this files.txt as a string to this IDL script?

  • 2
    seems to work here; are you doing something like cat file* | xargs ...? – Jeff Schaller Oct 15 '15 at 17:17
1

You could use a small script with a loop like this

Let it be runIDL.sh

#!/bin/ksh
while (read filepath)
do    
    /software/Linux/itt/idl/idl81/bin/idl -e program -args "$filepath"
done < "$1"

It's simple, elegant and should work like a charm. Just run ./runIDL files.txt and that's it!

-1

GNU Parallel may work for you:

cat files.txt | parallel /software/Linux/itt/idl/idl81/bin/idl -e program -args {}

It will default to 1 process per CPU core and it will make sure the output of two parallel jobs will not be mixed.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.