How to pass variable into parallel script from text file line by line

Question

I need to read a file line by line and pass the string as a variable for a script to be run in parallel.

ex:

14
43
57

is in foo.txt

if I var=$(cat foo.txt) then I can use this $var for inside a script, but how to call parallel to change this variable for each instance running the script?

Answer 1

If you really want a variable to be set:

parallel -a foo.txt 'myvar={}; myscript'

But otherwise I think Tom's solution will be what you want most of the time.

Answer 2

You can also use GNU Parallel with the -a flag, which reads from a file.

parallel -a foo.txt echo
14
43
57

It's not the most elegant example but it will just run the same command against all the arguments (line by line) in the text file. Include the -j flag to specify how many processes to run in parallel.

Answer 3

you can use xargs.

for example: the input - input.txt

1
2
3
4

The command to run command.sh

#/bin/bash
echo  $1: start
date
sleep 2s
echo  $1 : stop
date 

Running all together in parallel:

➜  /tmp  cat input.txt| xargs -n 1 -P 4 ./command.sh
1: start
3: start
2: start
4: start
Tue May 19 19:03:30 IDT 2015
Tue May 19 19:03:30 IDT 2015
Tue May 19 19:03:30 IDT 2015
Tue May 19 19:03:30 IDT 2015
1 : stop
4 : stop
3 : stop
2 : stop
Tue May 19 19:03:32 IDT 2015
Tue May 19 19:03:32 IDT 2015
Tue May 19 19:03:32 IDT 2015
Tue May 19 19:03:32 IDT 2015