3

I have some php scripts which I run in sequential order like:

php index.php import file1 --offline &&
php index.php import file2 --deleteUnused &&
php index.php import file3

Now I have newly discovered the parallel command and I tried something like that:

parallel -j 3 -- "php index.php import file1 --offline" "php index.php import file2 --deleteUnused" "php index.php import file3"

And it works perfectly.

Is it possible that I could have a file where all the above commands are included, and to start parallel with an option to read the commands from the file?

Something like this:

parallel -XX myFileWithCommands.txt
  • 1
    What parallel version are you using? Run parallel --version and mention the version in the question. I have "GNU parallel 20130522" for example. – Cristian Ciupitu Jul 2 '14 at 20:43
  • Now i get this: $ parallel --version parallel: invalid option -- '-' parallel [OPTIONS] command -- arguments for each argument, run command with argument, in parallel parallel [OPTIONS] -- commands run specified commands in parallel – MadeOfSport Jul 2 '14 at 20:50
  • Does strings which parallel` | fgrep -i gnu` output anything? – Cristian Ciupitu Jul 2 '14 at 20:52
  • 1
    My failure was: I installed moreutils - after installing sudo apt-get install parallel everything works fine. Sorry for that. – MadeOfSport Jul 2 '14 at 20:53
  • I thought so :-) – Cristian Ciupitu Jul 2 '14 at 20:55
5

Create myFileWithCommands.txt:

php index.php import file1 --offline
php index.php import file2 --deleteUnused
php index.php import file3

Then run parallel like this:

parallel -j 3 -- < myFileWithCommands.txt
  • wouldn't the double ampersand mean that every command will wait until the command before will be exited – MadeOfSport Jul 2 '14 at 20:19
  • @MadeOfSport, those are leftovers. I'll remove them. – Cristian Ciupitu Jul 2 '14 at 20:21
  • 1
    If you use GNU Parallel the '-- sh -c' is not needed – Ole Tange Jul 2 '14 at 20:27
  • @Cristian Ciupitu I think im to dumb to execute this.... parallel --gnu -j 3 --arg-sep " " -- sh -c < test.txt parallel: invalid option -- '-' parallel [OPTIONS] command -- arguments for each argument, run command with argument, in parallel parallel [OPTIONS] -- commands run specified commands in parallel – MadeOfSport Jul 2 '14 at 20:27
  • @MadeOfSport, drop --gnu, your version might not have or need it. – Cristian Ciupitu Jul 2 '14 at 20:31
2

Well, there are two simple ways to accomplish this that I can imagine. You can either pipe the contents of the file into parallel, or you can write a shell script. If all you want to accomplish is this particular set of tasks, then a shell script might make more sense. It would be very short, sweet and to-the-point as well:

#!/bin/sh
parallel -j 3 -- "php index.php import file1 --offline" "php index.php import file2 --deleteUnused" "php index.php import file3"

Now, if you want something more generalizable (i.e., something so that you can make several sets of commands to be loaded as a group), then it might be easier to come up with a simple file format and then pipe the contents of the file into parallel.

The simplest way this could be done is if you had the text include the verbatim commands you wish to be run.

For example, if this were your text file (test.txt):

"php index.php import file1 --offline" "php index.php import file2 --deleteUnused" "php index.php import file3"

Then, you could run the following and it might work fine:

$ cat test.txt | parallel -j 3 -- 

You could also experiment with having a slightly more complex file format (e.g., each command on a new line and not surrounded by quotes). If you were to go down that road, you would need to do some minor parsing on the file before piping it to parallel which could be accomplished pretty easily with, say for example awk.

However, if you decide to create a more flexible file format like that, it might be helpful to create a helper shell script (or function) that would save you from having to type awk '<whatever patterns you match> { do something }' | parallel -j3 -- each time.

  • 1
    this also works fine. i appreciate your help... – MadeOfSport Jul 2 '14 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.