3

I have the VM running with ubuntu(14.04.4). Sometimes, I just want to restart the VM by reboot command. When I execute the command, it prompts "success" but nothing happened.

I checked the syslog, no special logs found. Some services(like ssh) seem to be shut down.

My VM just run into docker hang issue(1.12.6) before reboot and I just removed the docker files under /var/lib/docker. The zombie processes still exist.

This VM can be restarted from VM web console. (I don't have the permission)

I want to know,

  • What can I do when this happen?
  • How can I get the clue for this issue, such as something found under /proc? reboot using "shutdown -r" which try to kill all the running processes, any condition causes "reboot" hangs?
  • After first "reboot" command, can I run "reboot -f" again to shut down the system? I didn't try it since it's not safe.

P.S.

Magic commands can be executed for this issue and the server reboot successfully and immediately,

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

I know reboot may do restart gracefully, such as sync dirty pages to disk, kill all running processes and so on.

When I run reboot, there were some zombie processes generated in special namespaces. Since the docker container removed, the namespaces also removed and only task_struct left in memory which needed to be released by wait4

The inheritance between processes were,

Processes after container removed,

  1. docker containered can be killed and then the children adopted by init 1
  2. entrypoint startup process can not be killed since the container removed

  3. zombie always existed since parent was left in memory without wait4

    • init 1(global ns) -> docker containered(global ns)(sleep) -> docker entrypoint startup(container ns)(sleep) -> java(container ns)(zombie)
|improve this question|||||
  • I would stop/kill everything manually, run a couple of sync commands, and reboot by whatever means available. – Rui F Ribeiro May 10 '18 at 7:31
  • Thanks @RuiFRibeiro. I'm interesting in the root cause since the manual operations may also hang. I want to see if this is related to docker zombie issue(Since zombie process can't be killed and only cleaned by init 1 or try to kill its parent. In this case, the parent can't be killed and init 1 doesn't adopt the zombie process directly. I doubt this but no evidence) or anything else. Any way to verify this? Sometimes I also saw I/O wait of CPU is high using top, but no process consumes I/O using iotop along with this issue. Currently, magic commands help me to do reboot. – qingdaojunzuo May 10 '18 at 8:25
  • I thought it was one off situation, so I take this is a recurring issue. Either I misread, or the question is not entirely clear on that. – Rui F Ribeiro May 10 '18 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.