6

I know how to run a command in all subdirectories that match a pattern like this (lets say I want to run gh repo sync):

for d in ./*/ ; do (cd "$d" && gh repo sync) ; done

And runs the commands in a subshell in each directory serially.

What I want is to run all the commands in parallel but wait in the main script for them to all finish.

I'm using Ubuntu 22.04. Is that possible?

2 Answers 2

12

GNU Parallel does this excellently. I've been using this structure daily for a long time:

parallel --wd '{}' 'gh repo sync' ::: ./*/.git/..

Similar test on my own system:

$ parallel --wd '{}' 'pwd' ::: ~/my\ projects/*/.git/..
/home/username/my projects/acard
/home/username/my projects/acp01
[125 more]
8
  • This looks awesome! Can I somehow see the output of the commands in the console?
    – traveh
    Commented Apr 24, 2023 at 16:48
  • A side note about this vs the other answer: If the commands rely on shared resources and you need to limit how many run at one time, parallel has -j to set a maximum number of concurrent jobs. Much more difficult to do that with the simple loop in the other answer, though that one doesn't require an additional dependency.
    – Izkata
    Commented Apr 24, 2023 at 18:27
  • 3
    @traveh, of course parallel preserves command output. Moreover, I think you'll be able to drop cd using --wd option. Commented Apr 24, 2023 at 20:05
  • 1
    You have my vote anyway, but why do you have && cd - at the end, just before the process exits? Commented Apr 25, 2023 at 7:15
  • 1
    @MarkSetchell I was writing on a phone without access to a Linux machine, so I overthunked it :)
    – l0b0
    Commented Apr 25, 2023 at 8:08
12

Run each subshell command in the background and then wait for them all to complete

for d in ./*/
do
    ( cd "$d" && gh repo sync ) &
done
wait
1
  • This is also a good answer, but I'm accepting the other one as it is better suited for my needs (sorry).
    – traveh
    Commented Apr 25, 2023 at 2:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .