6

Looking at different jobs running on a system with shared resources, it seems the nice values are being ignored. Many jobs with nice set at 19 are running at 100% cpu load while other many other jobs with nice set at 0 are running as low as 10% cpu load.
All of these processes are demanding and run on an idle system would max out every cpu given to it (e.g. NAMD).

I read here that

"...while [a nice] value is adjustable it can be ignored by the kernel's scheduler in Linux implementations."

Is this true? Is it possible that the kernel could be ignoring the nice value? It seems this is what is going on but how can I be certain? I do not want to make this into an issue with the sys admin without being more sure. I have read the related posts discussing How is nice working? and nice not really helping on Linux but these do not discuss not working with CPU loads.

Could it be that once a task is given resources, it will hold on to them for some time before reassigning them to the higher priority task? The low priority task has been running for days while the higher priority task repeatedly starts lots of short but demanding calculations which run for less than 10 minutes. Could it be that in between the short tasks the system gives resources to the low priority task which then holds onto them?

I believe the system I am experiencing this is on a StackIQ wrapped CentOS 6.5 installation (though I could easily be mistaken on some detail).

Improve this question

3 Answers 3

Reset to default
6

The nice value will not tell you anything about the actual cpu load, a process produces.

Nice-ness is just what you would think: the way, a process behaves under certain work loads.

To be more exact:

  • if a process with a high nice-value (==lower scheduling propability) is scheduled, it WILL hold the cpu, until a process with lower nice-value and/or higher priority requests the cpu and MAY create 100% load.

  • if a process with lower nice-value (==higher scheduling propability) relinquishes the cpu, it may not have used it at peak.

This is, why you see lower-niced processes using less cpu than higher-niced processes : The nicer process will give up more easily, but it obviously has more work to do at the moment ...

Improve this answer
1
  • This is how I understood nice to work but this is not what I am seeing. The processes with higher priority, nice value of 0, are not getting the CPUs while the processes with lower priority, nice value of 19, are getting the CPUs. I added to the post that the higher priority tasks are NAMD compute jobs which would max out all the CPUs given in the startup script but this is not happening, on several of the CPUs they are only running as low as 10% (some are at 100% but not many and I expected them all to be).
    – Steven C. Howell
    Commented Aug 11, 2015 at 14:50
4

A lot of processes in your system are just sitting and waiting for something to happen. This is what most server processors or daemons do (e.g. ssh, mail, X).

If you are working in a word processor, it just sits and waits until you click somewhere or type something. Most of the time the process is not scheduled and uses a very low amount of CPU time. You can set the nice-value for this process to zero and it won't have any noticeable effect.

If you however would be using a program to render 3-D high-resolution images or process animations or video, then the nice-value will have an effect.

Processes with shared resources face another problem: shared resources. They will often have to wait for the resource while another process is using it.

A hard disk is one example. If a process starts a hark disk operation (e.g. read or write a file), then it will be suspended by the system until the operation completes. Even a zero nice value won't help you there. While the process is suspended, it won't be using any CPU time.

The effect of nice can be very subtle ;-)

Improve this answer
3

10 minutes is very much long-term as far as Linux's scheduler is concerned. Time slices are something like 10ms.

When you're looking at CPU usage percentages, keep in mind that top adds up the per-thread usage of multi-threaded processes. So a 10-thread process that has each thread getting 10% active time will show up as using 100% of a CPU.

Linux's scheduler won't starve a nice 19 task (because deadlock bugs are hard to avoid if a process can be descheduled forever), so even nice 19 won't stop a task from getting some CPU time. If it has a lot of threads, it may still use significant CPU resources.

If some of the processes are blocking on I/O, especially virtual memory paging, their CPU usage % will go way down. Run something like dstat to see CPU usage breakdowns, disk, network, paging, and context switches. It's like vmstat but colourized and nicer.

Make sure your processes really are niced the way you think they are, by looking at the NI column in top. (It's unlikely that different threads in the same process will have different nice levels, but I think possible.)

If you've been using renice, remember that it's not recursive. renice-ing a parent process won't affect existing children, only future children.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .