3

1. Summary

I don't understand, how I can combine parallel and sequential commands in Linux.


2. Expected behavior

Pseudocode:

pip install pipenv sequential pipenv install --dev

parallel task

npm install -g grunt-cli sequential npm install

Windows batch working equivalent:

start cmd /C "pip install pipenv & pipenv install --dev"
start cmd /C "npm install -g grunt-cli & npm install"

3. Not helped

  1. I don't think, that & and wait can solve this problem, see rsaw comment.
  2. I read, that GNU parallel — is better way for parallel tasks, but I can't find, which syntax I need to use in GNU parallel, that solve this task.
  3. I try parallelshell:

    parallelshell "pip install pipenv && pipenv install --dev" "npm install -g grunt-cli && npm install"
    

    Full .sh file:

    git clone --depth 1 https://github.com/Kristinita/KristinitaPelican
    wait
    cd KristinitaPelican
    wait
    parallelshell "pip install pipenv && pipenv install --dev" "npm install -g grunt-cli && npm install"
    

    But at first pipenv install --dev command run for me, then npm install. It sequential, not parallel.

  • 1) Why do you have wait after each command in the .sh file, and 2) I've read rsaw's comment but please explain in your own words why wait does not work for what you're doing. – dsstorefile1 Mar 1 '18 at 7:57
2

Simply with GNU parallel:

parallel ::: 'pip install pipenv && pipenv install --dev' \
             'npm install -g grunt-cli && npm install'
2

The Unix-style equivalent of your Windows commands is the following:

(pip install pipenv && pipenv install --dev) >& pip.log &
(npm install -g grunt-cli && npm install) >& npm.log &

This starts two sub-shells in the background, one running pip followed by pipenv (if pip completes without error), the other running the two npm commands. Both sets of commands’ outputs are redirected to log files, to keep things intelligible.

If you need to wait for all those commands to complete before continuing, use

wait

as you’ve discovered already.

When sequencing and parallelising tasks using a POSIX-style shell, there are two tools at your disposal:

  • job control, which allows you to place tasks in the background (using & at the end of a command), and allows you to wait for their completion (using wait)
  • sub-shells, which allow you to split units of processing up in whatever way you like, including serialising tasks which need to be handled sequentially.

That’s good enough for simple requirements such as your current task, and can get you quite a long way. More advanced tools exist for more complex use-cases; GNU Parallel is one, it’s also worth looking into batching tools if you need finer resource control.

  • Thanks for the answer. But I still have a problems. // 1. If I run your script in Windows use Cygwin, pipenv doesn't install for me, see pip.log traceback. If I run commands sequential or I use Windows parallel tasks from my batch script above, all success. (I get same traceback, when I try npm parallel). – Саша Черных Mar 1 '18 at 9:44
  • // 2. Travis CI skip your script. If I use commands sequential, build passed. // Thanks. – Саша Черных Mar 1 '18 at 9:44
  • Weird, I would expect that to work on Travis CI... Never mind, since parallel works for you! – Stephen Kitt Mar 1 '18 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.