queue-like behaviour for multiple one-core jobs on single machine? [duplicate]

Question

I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many going at once the computer freezes (apparently 300 is okay, 400 too much...).

Ideally, what I'd like to do is run 24, then when one finishes, the 25th, then when the next finishes, the 26th, and so on. Unfortunately each job can take a different, and variable run time, so I can't do some kind of chron to set them going at staggered times.

I've seen some things with "wait", but I'm not sure if I sent 24 then, say, 976 with a wait command, would it give me the desired behaviour, or would it just run 976 in series after the first of the 24 finish?

EDIT: Thanks, this could very well be a duplicate, but as I see that question's answers only point towards parallel, can I please continue to explore here how to do it with xargs?

Reason for this, is that the Mac in question is currently on another continent and I absolutely need it to work for the next few days and run all these jobs - installing something always has the potential to mess up the machine, and so I don't want to install parallel at this point while I can't physically get to it. But it has xargs in bash, so I'm exploring using that.

Thus far, I've rewritten my bash script to meet what appears to be the situation expected by both xargs and parallel, that I can run it with a variety of input. So now, what I have is a bash script that runs my jobs on each file in a folder. I've currently tried:

ls -d myfolder/* | xargs -P 2 -L 1 ~/bin/myscript.sh

But this still seems to run them all simultaneously, thus I'm not sure what I've done wrong. (here I'm using max 2 just so I can keep looking and testing! I put only 4 in the folder - didn't want to send hundreds by accident)

FINAL EDIT: Ahah!!! MUCH later I figured out what I'd done wrong. xargs was likely running my script in parallel, but not the program I'd written the script to run. I wrote a script because I hadn't been able to figure out how to insert the filename into the arguments list, which expected parameter=value pairs. I eventually figured out how I could do this with the -I flag in xargs. This finally worked:

ls -d myfolder/* | xargs -I foo -P 2 -L 1 myprogram arg1 arg2 arg3=foo arg4

(I think -I and -L 1 are redundant, but as it works I'm not messing with it...)

Here, foo was replaced in the arguments list to myprogram with each filename. I note that one reason it took me ages to figure out is most instructions with -I use {} as the element to replace, and for some reason on my Macs it couldn't handle that. So I thought -I wasn't working, but it worked fine with foo.

Answer 1

There are a variety of tools that you can use, but I think the best is GNU parallel. It is considerably better than xargs in flexibility and utility. Linked is a description of GNU parallel for MacOS. I think you'll find this is likely the best solution for queuing as you seek.

Further information about using parallel is here.

Answer 2

I encountered a similar problem recently. As far as I know you have two options:

xargs -0 -P 24 -L 1

and

Gnu Parallel

---

For example, to convert every flac file found by the find command to ogg I tried running:

 find -name "*.flac" -print0 | xargs -0 -P 24 -L 1 oggenc

This runs up to -P 24 processes at a time using -L 1 lines from the find command. I'm sure you can use this to customize it to your needs but we will need more details from your question.