Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I work on a shared cluster. I've seen people run parallelized c code on this cluster which, when I use top to see what processes are running, are shown to be using (for example) 400% of the CPU, since they are using four processors for a single instance of their code.

Now someone is running (what I hear to be) a parallelized Python code. However, instead of top showing the Python code to be using 400% of the CPU, it is being shown as four different processes, each using their own processor (at 100%).

I am wondering, does Python (when parallelized) show with top as running as many different processes (as opposed to C) or is this Python code not actually running in parallel?

I don't know if Stack Exchange would be a better place for this question. Since I am using top I figured this place would be better. Let me know if I should move it.

share|improve this question
1  
that depends on the Python library that's being used for parallelization. If it's the multiprocessing module it uses processes for parallelization so no surprise you are seeing what you are seeing –  1_CR Jun 14 at 0:05
    
Is it possible that that module would cause Python to implement twice the number of processes than there are processors available? That is currently what is happening. –  Joshua Jun 14 at 0:09
1  
the multiprocessing module allows one to create a process pool with an arbitrary of helper processes. The default value is equal to the number of processors on the box but this can be programmatically overridden –  1_CR Jun 14 at 0:13

2 Answers 2

up vote 3 down vote accepted

What you see in C is using threads, so the process usage is the total of all its threads. If there are 4 threads with 100% CPU usage each, the process will show as 400%

What you see in python is almost certainly parallelism via the multiprocess model. That's a model meant to overcome Python's threading limitations. Python can only run itself one thread at a time (see the Python Global Interpreter Lock - GIL). In order to do better than that one can use the multiprocess module which ends up creating processes instead of threads, which in turn show in ps as multiple processes, which then can use up to 100% CPU each since they are (each) single-threaded.

I bet that if you run ps -afeT you'll see the threads of the C program but no additional threads for the python program.

share|improve this answer

The reference Python implementation, CPython, has the Global Interpreter Lock (GIL), which prevents it from running code in parallel, only concurrently. Threading is only useful for I/O. You need multiple processes to be able to execute in parallel. The C code you are looking at is likely using threading for parallelism instead.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.