How to use GNU Parallel for executing a program concurrently?

Question

I need to run a python script several times in parallel but I have done executing it in the background like this

ipython program.py & ipython program.py & ...

I want to know if this way uses one core per execution or just executes the program.py using threads. By the way, I want to explore the use of GNU Parallel but the examples that I find are about commands like "cat" of "find". How can I use GNU Parallel for executing program.py concurrently, each time in a different core? Thanks for your help.

Answer 1

How can I use GNU Parallel for executing program.py concurrently, each time in a different core?

You (almost) never want to peg a program to a certain core. Typically you do not care which core is doing the work. And often you simply want to run one job for each CPU thread in the system.

And that is easy to do using GNU Parallel:

seq 1000 | parallel ipython program.py

This will run ipython program.py 1 .. ipython program.py 1000 but only run one job per CPU thread in parallel. So on an 8 core machine with hyperthreading (i.e. 16 CPU threads) it will start 16 jobs in parallel.

This covered in chapter 2 of https://doi.org/10.5281/zenodo.1146014 which I encourage you to spend 15 minutes reading. Your command line wiĺl love you for it.

Answer 2

I want to know if this way uses one core per execution

No, not necessarily,

or just executes the program.py using threads

no, these are separate processes, not threads. And threads could be scheduled on the same or different cores, too.

So, you seem to be mixing up a few terms.

What your operating system does when you start processes is schedule them, which means they get executed on a free CPU core, when one is available. It's pretty clever about that, so most likely your different python processes will be run on different cores – but not necessarily (this works just as well if you have just 1 core!).

How can I use GNU Parallel for executing program.py concurrently, each time in a different core?

Not at all, that's not how multiprocessor scheduling works (without going through tricks like core pinning), and it would have no advantage, quite likely: If your programs don't already fully utilize your CPU cores the way you are already calling them, GNU parallels won't change that. If not all cores are used automatically, it means that your processes are competing for other resources than CPU time.

Quite likely, for example, they are relatively storage bandwidth-intense, as you seem to be using ipython (which loads a lot of Python modules that you don't need unless you're actually running interactively, which you can't, since you're starting them all in parallel) instead of plain python.

Generally, I find GNU parallel quite unwieldy, i.e., unnecessarily hard to use. I find that echo argument1 argument2 argument3 | xargs -P0 -n1 python script.py is a much easier way to start 3 three processes (python script.py argument1, python script.py argument2 and python script.py argument3) in parallel.