Cloud Shell idle cpu usage seems relatively high (3-6%) - where is this usage coming from?

Question

top - 12:10:09 up  2:18,  6 users,  load average: 0.03, 0.03, 0.00
Tasks:  35 total,   1 running,  34 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.1 us,  1.4 sy,  0.0 ni, 96.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1735912 total,   437708 free,   378916 used,   919288 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1188436 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                            
  251 alan_ch+  20   0  636432  12428   6504 S  0.2  0.7   0:02.46 python         
    1 root      20   0   17980   2800   2496 S  0.0  0.2   0:00.01 bash           
    8 root      20   0  250120   2212   1776 S  0.0  0.1   0:00.84 rsyslogd
   38 root      20   0   69952   3040   2268 S  0.0  0.2   0:00.00 sshd                                                               

Google Cloud includes Cloud Shell. This provides an interactive Linux sandbox, which can be used for light administration or testing tasks. It is available without payment or payment information. Terms and conditions apply.

The summary CPU usage in top consistently shows as around 3-6%, even after increasing the interval (top -d 10). But, the if you look at the CPU usage shown for the individual processes, it doesn't add up.

What is using the CPU?

Answer 1

The current Cloud Shell creates a private VM, for maximum flexibility. But the actual shell it shows you is running inside a container. The container is very privileged, but since it uses a separate PID namespace you cannot see the processes outside the container.

$ amicontained
Container Runtime: kube
Has Namespaces:
        pid: true
        user: false
AppArmor Profile: unconfined
Capabilities:
        BOUNDING -> chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_override mac_admin syslog wake_alarm block_suspend audit_read
Chroot (not pivot_root): false
Seccomp: disabled

Privileged containers can be escaped, using a few carefully crafted shell commands. This is not a security exploit, it's just not the intended use. (Do not abuse the Cloud Shell resources. You can trust that Google know something about monitoring resource usage of virtual machines).

E.g. I was able to mount devtmpfs, create some pipes, leverage core_pattern to open a shell on /dev/fifo_in / /dev/fifo_out. Inside the shell, I re-entered the containers mount namespace only. Then it is sufficient to over-mount /proc with a new procfs mount. With that, I could open top in another Cloud Shell tab, and it would show the host PID namespace.

Let's say they haven't (yet) optimized the idle resource usage.

top - 12:50:14 up  2:58,  3 users,  load average: 0.01, 0.05, 0.04
Tasks: 127 total,   1 running,  96 sleeping,   0 stopped,   1 zombie
%Cpu(s):  3.1 us,  1.5 sy,  0.0 ni, 95.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1735912 total,   444240 free,   351448 used,   940224 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1215836 avail Mem

USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                  
root      20   0  413140  58704  26008 S  2.7  3.4   4:31.18 kubelet      
root      20   0  378940  46260  15912 S  0.6  2.7   1:03.18 dockerd                                                                  
root      20   0  184008   9760   5408 S  0.1  0.6   0:14.27 python         
alan_ch+  20   0  644372  12392   6584 S  0.1  0.7   0:00.07 python         
root      20   0   66364  33976  33472 S  0.1  2.0   0:10.34 systemd-journal   
root      20   0  292264  22488  13616 S  0.1  1.3   0:04.56 docker-containe   
root      20   0  369808  64224  36928 S  0.1  3.7   0:03.81 dockerd          
root      20   0  365684   6408   3668 S  0.1  0.4   0:06.00 docker-containe
alan_ch+  20   0   58060  20272   7560 S  0.0  1.2   0:01.27 python                                                                   
root      20   0       0      0      0 I  0.0  0.0   0:04.42 rcu_sched