1

I need to run a command for each individual instance of a given variable name in parallel. Sometimes, there might be 4 variables, other times there might be 100. For example, say I have this particular dataset as:

datanames='KQPW KMMX KMKO KZAO'

I need to run a process for each which is to be run in parallel with one-another. In other words, I need to run process1 for KQPW while running process1 for KMMX while ... etc. Process1 requires input based on the variablename.

From the tutorials I have read, and some initial digging, I have installed the GNU 'parallel' command. I have put all of the datanames into a textfile called "run.txt":

KQPW.csh
KMMX.csh
KMKO.csh
KZAO.csh

wherein each of the .csh files contains the command for calling process1 with the unique variable name as the necessary input to process1. The question is, how do I run all four of these commands at once? I tried:

cat run.txt | parallel

but nothing happened. Any thoughts?

0

A quick demonstration of executing scripts based as described in parallel, without using any external tools:

#!/bin/bash
datanames='KQPW KMMX KMKO KZAO'
datanamesarray=($datanames)
for item in ${datanamesarray[@]}; do
  ( ./${item}.csh; sleep 10 ) &
done
echo waiting..
wait
echo done

Executing this will display waiting.. followed by a ten second delay as all of the subshells are executed in parallel. wait will pause the parent script until all subshells have terminated before proceeding. The echo, sleep, and wait statements are here for demonstrative purposes.

6
  • How is that better than for item in $datanames;? Apr 11 '18 at 18:25
  • That will also work in this case, but I happen to prefer to iterate over an array, so I reflexively converted your space-delimeted string.
    – DopeGhoti
    Apr 11 '18 at 18:27
  • So the issue is that it is running everything still at once.. Apr 11 '18 at 19:45
  • I don't understand. I thought you wanted everything to run in parallel?
    – DopeGhoti
    Apr 11 '18 at 20:19
  • i think this method only runs a single job each time, so this is not really running parallel
    – ikel
    Jan 4 '20 at 18:41
0

What you are doing should work. If you can do:

$ KMMX.csh

then GNU Parallel should also be able to run the script. The first debugging step is to run parallel with --dry-run, which will tell you what is being run:

cat run.txt | parallel --dry-run

There are, however, several things in your question that puzzles me:

  • The title of the question mentions bash but you call your scripts *.csh Is your interactive shell bash but the scripts csh-files?
  • You do not mention that you have put the files in a dir in your $PATH, yet you do not refer to the scripts with path (e.g. ./KMMX.csh). That could explain why it does not work. Though you ought to get an error message.
  • You do not mention that you have chmod +x the files. That could explain why it does not work. Though you ought to get an error message.

But most of all I worry that you are doing this harder on yourself than it needs to be. GNU Parallel is designed to build commands from a template, and it sounds as if your *.csh-files are also built from a template with the only change being the dataname.

My guess is that you can change your setup to:

$ parallel doit {} ::: KQPW KMMX KMKO KZAO

where doit is a script/function that takes the dataname as an argument.

I will highly encourage you to watch the intro videos for a quick introduction: http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 Then look at the EXAMPLEs after the list of OPTIONS (Use LESS=+/EXAMPLE: man parallel). That will give you an idea of what GNU parallel is capable of. Then spend a couple of hours walking through the tutorial (man parallel_tutorial).

Your command line will love you for it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.