I need to run a python script several times in parallel but I have done executing it in the background like this

ipython program.py & ipython program.py & ...

I want to know if this way uses one core per execution or just executes the program.py using threads. By the way, I want to explore the use of GNU Parallel but the examples that I find are about commands like "cat" of "find". How can I use GNU Parallel for executing program.py concurrently, each time in a different core? Thanks for your help.

  • You are starting several programs almost simultaneously; depending on what else is running it's possible they will run in different cores. However, the OS will try to make best use of the hardware on a system-wide basis. Dec 2, 2021 at 18:13
  • ipython is meant to be used as interactive shell, not for batch processing. You probably just want to use python. Dec 2, 2021 at 19:18

2 Answers 2


How can I use GNU Parallel for executing program.py concurrently, each time in a different core?

You (almost) never want to peg a program to a certain core. Typically you do not care which core is doing the work. And often you simply want to run one job for each CPU thread in the system.

And that is easy to do using GNU Parallel:

seq 1000 | parallel ipython program.py

This will run ipython program.py 1 .. ipython program.py 1000 but only run one job per CPU thread in parallel. So on an 8 core machine with hyperthreading (i.e. 16 CPU threads) it will start 16 jobs in parallel.

This covered in chapter 2 of https://doi.org/10.5281/zenodo.1146014 which I encourage you to spend 15 minutes reading. Your command line wiĺl love you for it.

  • Thanks for your answer. Now I understand that I can't control the uses of the cores. I just send the jobs and the computer decides how to use them. Well, it doesn't matter because I just need to execute the program several times in parallel. Your answer worked very well.
    – Fabio
    Dec 6, 2021 at 15:00
  • @Fabio Technically you can control which core is used using taskset, but it is extremely rare that you need that. Normally you simply let the OS decide.
    – Ole Tange
    Dec 6, 2021 at 17:00

I want to know if this way uses one core per execution

No, not necessarily,

or just executes the program.py using threads

no, these are separate processes, not threads. And threads could be scheduled on the same or different cores, too.

So, you seem to be mixing up a few terms.

What your operating system does when you start processes is schedule them, which means they get executed on a free CPU core, when one is available. It's pretty clever about that, so most likely your different python processes will be run on different cores – but not necessarily (this works just as well if you have just 1 core!).

How can I use GNU Parallel for executing program.py concurrently, each time in a different core?

Not at all, that's not how multiprocessor scheduling works (without going through tricks like core pinning), and it would have no advantage, quite likely: If your programs don't already fully utilize your CPU cores the way you are already calling them, GNU parallels won't change that. If not all cores are used automatically, it means that your processes are competing for other resources than CPU time.

Quite likely, for example, they are relatively storage bandwidth-intense, as you seem to be using ipython (which loads a lot of Python modules that you don't need unless you're actually running interactively, which you can't, since you're starting them all in parallel) instead of plain python.

Generally, I find GNU parallel quite unwieldy, i.e., unnecessarily hard to use. I find that echo argument1 argument2 argument3 | xargs -P0 -n1 python script.py is a much easier way to start 3 three processes (python script.py argument1, python script.py argument2 and python script.py argument3) in parallel.

  • Thanks for your explanation. I was mixing the concepts of Process and Threads. Now, if I understood well, the computer decides if use one core with several processes concurently or different cores. ¿Right? So, I can't control the use of the cores.
    – Fabio
    Dec 6, 2021 at 14:46
  • that's right, it decides that by itself. You can take influence, but it's very rare that you'll find a situation where you run different python processes, and you'd be "smarter" than your computer about assigning cores to processes. Dec 6, 2021 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.