I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many going at once the computer freezes (apparently 300 is okay, 400 too much...).

Ideally, what I'd like to do is run 24, then when one finishes, the 25th, then when the next finishes, the 26th, and so on. Unfortunately each job can take a different, and variable run time, so I can't do some kind of chron to set them going at staggered times.

I've seen some things with "wait", but I'm not sure if I sent 24 then, say, 976 with a wait command, would it give me the desired behaviour, or would it just run 976 in series after the first of the 24 finish?

EDIT: Thanks, this could very well be a duplicate, but as I see that question's answers only point towards parallel, can I please continue to explore here how to do it with xargs?

Reason for this, is that the Mac in question is currently on another continent and I absolutely need it to work for the next few days and run all these jobs - installing something always has the potential to mess up the machine, and so I don't want to install parallel at this point while I can't physically get to it. But it has xargs in bash, so I'm exploring using that.

Thus far, I've rewritten my bash script to meet what appears to be the situation expected by both xargs and parallel, that I can run it with a variety of input. So now, what I have is a bash script that runs my jobs on each file in a folder. I've currently tried:

ls -d myfolder/* | xargs -P 2 -L 1 ~/bin/myscript.sh

But this still seems to run them all simultaneously, thus I'm not sure what I've done wrong. (here I'm using max 2 just so I can keep looking and testing! I put only 4 in the folder - didn't want to send hundreds by accident)

FINAL EDIT: Ahah!!! MUCH later I figured out what I'd done wrong. xargs was likely running my script in parallel, but not the program I'd written the script to run. I wrote a script because I hadn't been able to figure out how to insert the filename into the arguments list, which expected parameter=value pairs. I eventually figured out how I could do this with the -I flag in xargs. This finally worked:

ls -d myfolder/* | xargs -I foo -P 2 -L 1 myprogram arg1 arg2 arg3=foo arg4

(I think -I and -L 1 are redundant, but as it works I'm not messing with it...)

Here, foo was replaced in the arguments list to myprogram with each filename. I note that one reason it took me ages to figure out is most instructions with -I use {} as the element to replace, and for some reason on my Macs it couldn't handle that. So I thought -I wasn't working, but it worked fine with foo.

  • Also GNU xargs with --max-procs – muru Jul 27 '19 at 14:47
  • A page or two of Python using the multiprocessing package would do the trick. Set the number of workers to however many you want to run simultaneously and enqueue the jobs to run as they will. – stolenmoment Jul 27 '19 at 14:47

I encountered a similar problem recently. As far as I know you have two options:

xargs -0 -P 24 -L 1


Gnu Parallel

For example, to convert every flac file found by the find command to ogg I tried running:

 find -name "*.flac" -print0 | xargs -0 -P 24 -L 1 oggenc

This runs up to -P 24 processes at a time using -L 1 lines from the find command. I'm sure you can use this to customize it to your needs but we will need more details from your question.

  • Thanks - I've edited my question more, and I've tried: ls -d myfolder/* | xargs -P 2 -L 1 ~/bin/myscript.sh but it still runs them all at once (changing to 2 from 24 just for testing). It does not look like the -0 option is relevant. Is there something about ls I need to modify? – Vas Jul 29 '19 at 16:43
  • @Vas ls gives all the files on one line. You need to seperate them somehow so that xargs can run the commands individually. Try: find -type d myfolder -print0 – SurpriseDog Jul 29 '19 at 20:09
  • Hmm, I've tried find myfolder -type f -print0 | xargs -0 -P 2 -L 1 ~/bin/myscript.sh and it just did them all again. (-type f as I want to run it on the files inside the folder). – Vas Jul 29 '19 at 21:44
  • Ahah!!! MUCH later, I figured out what I did wrong. I had written a script to run my programme, and it was running probably the script in parralel, but not the program ist – Vas Sep 6 '19 at 10:52
  • *the program itself. More details in edit above... – Vas Sep 6 '19 at 11:07

There are a variety of tools that you can use, but I think the best is GNU parallel. It is considerably better than xargs in flexibility and utility. Linked is a description of GNU parallel for MacOS. I think you'll find this is likely the best solution for queuing as you seek.

Further information about using parallel is here.

  • your first link goes nowhere? – frostschutz Jul 28 '19 at 12:08
  • 1
    Fixed the link. Unsure as to why that happened, but that is the original! Thanks. – trailhounds Jul 29 '19 at 19:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.