3

1. Summary

I don't understand, how I can combine parallel and sequential commands in Linux.


2. Expected behavior

Pseudocode:

pip install pipenv sequential pipenv install --dev

parallel task

npm install -g grunt-cli sequential npm install

Windows batch working equivalent:

start cmd /C "pip install pipenv & pipenv install --dev"
start cmd /C "npm install -g grunt-cli & npm install"

3. Not helped

  1. I don't think, that & and wait can solve this problem, see rsaw comment.
  2. I read, that GNU parallel — is better way for parallel tasks, but I can't find, which syntax I need to use in GNU parallel, that solve this task.
  3. I try parallelshell:

    parallelshell "pip install pipenv && pipenv install --dev" "npm install -g grunt-cli && npm install"
    

    Full .sh file:

    git clone --depth 1 https://github.com/Kristinita/KristinitaPelican
    wait
    cd KristinitaPelican
    wait
    parallelshell "pip install pipenv && pipenv install --dev" "npm install -g grunt-cli && npm install"
    

    But at first pipenv install --dev command run for me, then npm install. It sequential, not parallel.

0

2 Answers 2

3

Simply with GNU parallel:

parallel ::: 'pip install pipenv && pipenv install --dev' \
             'npm install -g grunt-cli && npm install'
2
2

The Unix-style equivalent of your Windows commands is the following:

(pip install pipenv && pipenv install --dev) >& pip.log &
(npm install -g grunt-cli && npm install) >& npm.log &

This starts two sub-shells in the background, one running pip followed by pipenv (if pip completes without error), the other running the two npm commands. Both sets of commands’ outputs are redirected to log files, to keep things intelligible.

If you need to wait for all those commands to complete before continuing, use

wait

as you’ve discovered already.

When sequencing and parallelising tasks using a POSIX-style shell, there are two tools at your disposal:

  • job control, which allows you to place tasks in the background (using & at the end of a command), and allows you to wait for their completion (using wait)
  • sub-shells, which allow you to split units of processing up in whatever way you like, including serialising tasks which need to be handled sequentially.

That’s good enough for simple requirements such as your current task, and can get you quite a long way. More advanced tools exist for more complex use-cases; GNU Parallel is one, it’s also worth looking into batching tools if you need finer resource control.

3
  • Thanks for the answer. But I still have a problems. // 1. If I run your script in Windows use Cygwin, pipenv doesn't install for me, see pip.log traceback. If I run commands sequential or I use Windows parallel tasks from my batch script above, all success. (I get same traceback, when I try npm parallel). Mar 1, 2018 at 9:44
  • // 2. Travis CI skip your script. If I use commands sequential, build passed. // Thanks. Mar 1, 2018 at 9:44
  • Weird, I would expect that to work on Travis CI... Never mind, since parallel works for you! Mar 1, 2018 at 13:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .