0

We have got a new Linux el6.x86_64 server and after moving all our jobs (primarily java/C++/python/shell) processes onto this new server, we have observed the processes running slow. E.g. A python script running on previous server would finish in 30 minutes, on new server it takes 4 hours to process.

Below is the CPU usage statistics (captured with top) from new server, which shows only one task active at a time. I need your help to understand following:

  • If it has 32CPUs, shouldn't there be 32 tasks running in parallel?
  • All these CPUs are online, verified from /sys/devices/system/cpu. Could there be any other system setting CPU availability very low?
 
top - 02:04:04 up 5 days,  6:56, 32 users,  load average: 3.63, 3.91, 3.47
Tasks: 1208 total,   1 running, 1207 sleeping,   0 stopped,   0 zombie
Cpu0  : 23.4%us,  3.0%sy,  0.0%ni, 73.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 15.4%us,  5.6%sy,  0.0%ni, 79.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 12.2%us,  1.0%sy,  0.0%ni, 86.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  3.9%us,  0.3%sy,  0.0%ni, 95.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 19.3%us,  2.0%sy,  0.0%ni, 78.4%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  :  6.9%us,  0.3%sy,  0.0%ni, 92.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  1.0%us,  0.3%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu16 : 42.2%us,  4.0%sy,  0.0%ni, 53.5%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu17 :  0.7%us,  0.7%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu18 : 26.0%us,  2.0%sy,  0.0%ni, 71.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu19 :  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu20 : 53.8%us,  4.0%sy,  0.0%ni, 41.2%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu21 : 29.3%us,  1.3%sy,  0.0%ni, 69.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu22 :  2.0%us,  0.7%sy,  0.0%ni, 97.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu23 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu24 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu25 :  0.7%us,  1.0%sy,  0.0%ni, 98.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu26 :  0.7%us,  1.3%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu27 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu28 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu29 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu30 :  2.6%us,  0.0%sy,  0.0%ni, 97.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu31 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  396867664k total, 391584344k used,  5283320k free,   414764k buffers

  • DId you upgrade your OS to new version of OS?When you did OS upgrading.You should use new version of python.You needed to check your existing python script when run it on new python version.I suspected that can cause the process slowdown. – supriady Jan 6 '17 at 11:57
  • When You didnt upgrade OS.You should follow all old server configuration on new server configuration. – supriady Jan 6 '17 at 11:59
  • The OS and python versions are same. We are using Red Hat 4.4.7-15 with python 2.6.6. The installation is performed by SA team and they refuse to investigate if there are any differences on servers unless we could point out something is wrong with any conf. – user3305937 Jan 6 '17 at 12:28
3

The processes will not utilize the extra cores to run faster unless they are designed for it by using threads or other mechanisms for parallelism.

Python in particular is typically bad at exploiting multiple cores due to the GIL (Global interpreter Lock). You could try to use the 'multiprocessing' Python module to achieve more preferment software.

There is not much you can do to increase performance of single-core, not I/O bound processes, except buying CPUs with higher clock speeds. But sometimes you can find memory access and memory caching bottlenecks. For example, if your machine contains multiple CPUs that are connected together over NUMA connections, and processes move frequently between those CPUs, the situation may arise where CPUs have to frequently access memory over the NUMA connections through other CPUs (Memory connected to another CPU can be significantly slower to access then the memory connected directly to 'this' CPU).

It might be worthwhile to try and pin processes to particular CPUs by using the cpuset(1) command. If this works, you should also look into the various Kenel NUMA behaviour tuning parameters.

Edit to add: Another area to look at is power management, perhaps the CPU frequencies are getting scaled down because of power saving processes. You can use the cpupower(1) command to check and set VPU power management settings.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.