I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many going at once the computer freezes (apparently 300 is okay, 400 too much...).

Ideally, what I'd like to do is run 24, then when one finishes, the 25th, then when the next finishes, the 26th, and so on. Unfortunately each job can take a different, and variable run time, so I can't do some kind of chron to set them going at staggered times.

I've seen some things with "wait", but I'm not sure if I sent 24 then, say, 976 with a wait command, would it give me the desired behaviour, or would it just run 976 in series after the first of the 24 finish?

EDIT: Thanks, this could very well be a duplicate, but as I see that question's answers only point towards parallel, can I please continue to explore here how to do it with xargs?

Reason for this, is that the Mac in question is currently on another continent and I absolutely need it to work for the next few days and run all these jobs - installing something always has the potential to mess up the machine, and so I don't want to install parallel at this point while I can't physically get to it. But it has xargs in bash, so I'm exploring using that.

Thus far, I've rewritten my bash script to meet what appears to be the situation expected by both xargs and parallel, that I can run it with a variety of input. So now, what I have is a bash script that runs my jobs on each file in a folder. I've currently tried:

ls -d myfolder/* | xargs -P 2 -L 1 ~/bin/myscript.sh

But this still seems to run them all simultaneously, thus I'm not sure what I've done wrong. (here I'm using max 2 just so I can keep looking and testing! I put only 4 in the folder - didn't want to send hundreds by accident)

FINAL EDIT: Ahah!!! MUCH later I figured out what I'd done wrong. xargs was likely running my script in parallel, but not the program I'd written the script to run. I wrote a script because I hadn't been able to figure out how to insert the filename into the arguments list, which expected parameter=value pairs. I eventually figured out how I could do this with the -I flag in xargs. This finally worked:

ls -d myfolder/* | xargs -I foo -P 2 -L 1 myprogram arg1 arg2 arg3=foo arg4

(I think -I and -L 1 are redundant, but as it works I'm not messing with it...)

Here, foo was replaced in the arguments list to myprogram with each filename. I note that one reason it took me ages to figure out is most instructions with -I use {} as the element to replace, and for some reason on my Macs it couldn't handle that. So I thought -I wasn't working, but it worked fine with foo.

  • Also GNU xargs with --max-procs
    – muru
    Jul 27, 2019 at 14:47
  • A page or two of Python using the multiprocessing package would do the trick. Set the number of workers to however many you want to run simultaneously and enqueue the jobs to run as they will. Jul 27, 2019 at 14:47

2 Answers 2


There are a variety of tools that you can use, but I think the best is GNU parallel. It is considerably better than xargs in flexibility and utility. Linked is a description of GNU parallel for MacOS. I think you'll find this is likely the best solution for queuing as you seek.

Further information about using parallel is here.

  • your first link goes nowhere? Jul 28, 2019 at 12:08
  • 1
    Fixed the link. Unsure as to why that happened, but that is the original! Thanks. Jul 29, 2019 at 19:24

I encountered a similar problem recently. As far as I know you have two options:

xargs -0 -P 24 -L 1


Gnu Parallel

For example, to convert every flac file found by the find command to ogg I tried running:

 find -name "*.flac" -print0 | xargs -0 -P 24 -L 1 oggenc

This runs up to -P 24 processes at a time using -L 1 lines from the find command. I'm sure you can use this to customize it to your needs but we will need more details from your question.

  • Thanks - I've edited my question more, and I've tried: ls -d myfolder/* | xargs -P 2 -L 1 ~/bin/myscript.sh but it still runs them all at once (changing to 2 from 24 just for testing). It does not look like the -0 option is relevant. Is there something about ls I need to modify?
    – Vas
    Jul 29, 2019 at 16:43
  • @Vas ls gives all the files on one line. You need to seperate them somehow so that xargs can run the commands individually. Try: find -type d myfolder -print0 Jul 29, 2019 at 20:09
  • Hmm, I've tried find myfolder -type f -print0 | xargs -0 -P 2 -L 1 ~/bin/myscript.sh and it just did them all again. (-type f as I want to run it on the files inside the folder).
    – Vas
    Jul 29, 2019 at 21:44
  • Ahah!!! MUCH later, I figured out what I did wrong. I had written a script to run my programme, and it was running probably the script in parralel, but not the program ist
    – Vas
    Sep 6, 2019 at 10:52
  • *the program itself. More details in edit above...
    – Vas
    Sep 6, 2019 at 11:07

Not the answer you're looking for? Browse other questions tagged .