Why are there so few "running" processes?

Question

On all of BSD/Linux/MacOS and Solaris, I made an observation - no matter how busy the system is, top always reports very few (usually 1-2) running processes.

According to my limited understanding, a process is in "running" state if it is using CPU time, or waiting for CPU time to become available.

So, here is a scenario: I use SSH to tunnel VNC, then run top on remote machine, in a Terminal emulator:

• top itself is listed as a "running" process (understandable)
• sshd is listed, but not "running"
• Xvnc is listed, but not "running"

But sshd is tunneling VNC traffic, Xvnc is what keeps me updated with screen content, how can they not be using CPU time?

If they are using CPU time, then how come they are not "running"?

Answer 1

I believe this is a question of timing. top checks CPU state every N seconds. At the moment the check is run, top will have to be active since it is actively checking the state of the system. A nice way of seing this in action is to decrease the update time of top and watch your processes move! As suggested in man top:

   o  The  user  interface,  through  prompts and help, intentionally
      implies that the delay interval is limited to tenths of a  sec‐
      ond.   However,  you're  free to set any desired delay.  If you
      want to see Linux at his scheduling best, try a  delay  of  .09
      seconds or less.

      For this experiment, under x-windows open an xterm and maximize
      it.  Then do the following:
        . provide a scheduling boost and tiny delay via:
            nice -n -10 top -d.09
        . keep sorted column highlighting Off so as to
          minimize path length
        . turn On reverse row highlighting for emphasis
        . try various sort columns (TIME/MEM work well),
          and normal or reverse sorts to bring the most
          active processes into view

      What you'll see is a very busy Linux  doing  what  he's  always
      done  for you, but there was no program available to illustrate
      this.

If you try this, you should see more processes going into R mode. You can also try running top in batch mode a few times and collecting the processes that are marked as running:

sudo nice -n -10 top -d.09 -bn 100 | awk '$8~/R/{print $NF}' | sort -u

On my system under its current load (which includes an open ssh tunnel) I get the following processes:

cinnamon
firefox
kworker/0:1
kworker/1:2
kworker/2:2
kworker/3:0
plugin-containe
ssh
tint2
top
Xorg

Answer 2

Most processes are waiting for something to happen, a packet coming from the network, a disk I/O to complete, a human typing on its keyboard or exercising its mouse these days, with such fast CPUs being the norm.

Launch a compute intensive process, like some picture or movie processing and it will show up as running.

Of course, top, while analyzing the processes will always find itself running for obvious reasons.

Note that the status displayed is not an average value but a snapshot so even processes much busier than top might always be shown a sleeping for sampling reasons.