5

I have a small server with Ubuntu 10.04 on it; I am manipulating this server from another computer via ssh, and I tried to use nfs on it to share files. That mostly works, until one of the clients unmounts and I want to shutdown nfs-kernel-server on the server. While the stopping seems proper:

$ sudo service nfs-kernel-server stop
 * Stopping NFS kernel daemon                                                     [ OK ] 
 * Unexporting directories for NFS kernel daemon...                               [ OK ] 

... I do get something like this in the log:

Feb  5 11:50:17 user init: statd main process (3806) killed by KILL signal
Feb  5 11:50:17 user init: statd main process ended, respawning
Feb  5 11:50:17 user init: idmapd main process (3808) killed by KILL signal
Feb  5 11:50:17 user init: idmapd main process ended, respawning
Feb  5 11:50:17 user statd-pre-start: local-filesystems started
Feb  5 11:50:17 user sm-notify[3815]: Already notifying clients; Exiting!
Feb  5 11:50:17 user rpc.statd[3830]: Version 1.1.6 Starting
Feb  5 11:50:17 user rpc.statd[3830]: Flags: 

... meaning that some related processes to nfs didn't care about me saying stop, and respawned again. If at this point I try to do sudo service nfs-kernel-server start (again via ssh), that command freezes, and in /var/log/syslog I get this:

Feb  5 11:43:55 user mountd[2045]: authenticated mount request from 192.168.0.2:1005 for /media/disk (/media/disk)
Feb  5 11:45:19 user mountd[2045]: Caught signal 15, un-registering and exiting.
Feb  5 11:45:19 user kernel: [27428.148368] nfsd: last server has exited, flushing export cache
Feb  5 11:45:19 user kernel: [27428.148431] BUG: Dentry d0bc8b28{i=1f6,n=} still in use (1) [unmount of vfat sdd8]
Feb  5 11:45:19 user kernel: [27428.148473] ------------[ cut here ]------------
Feb  5 11:45:19 user kernel: [27428.148481] kernel BUG at /build/buildd/linux-2.6.32/fs/dcache.c:670!
Feb  5 11:45:19 user kernel: [27428.148491] invalid opcode: 0000 [#1] SMP 
Feb  5 11:45:19 user kernel: [27428.148501] last sysfs file: /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq
...
Feb  5 11:45:19 user kernel: [27428.148807] Call Trace:
Feb  5 11:45:19 user kernel: [27428.148824]  [<c024c780>] ? vfs_quota_off+0x0/0x20
Feb  5 11:45:19 user kernel: [27428.148838]  [<c021d4fc>] ? shrink_dcache_for_umount+0x3c/0x50
Feb  5 11:45:19 user kernel: [27428.148852]  [<c020d090>] ? generic_shutdown_super+0x20/0xe0
...
Feb  5 11:45:19 user kernel: [27428.149511] EIP: [<c021d4a9>] shrink_dcache_for_umount_subtree+0x249/0x260 SS:ESP 0068:ccc6de6c
Feb  5 11:45:19 user kernel: [27428.149631] ---[ end trace 6198103bb62887ac ]---
Feb  5 11:49:53 user init: idmapd main process (838) killed by TERM signal
Feb  5 11:49:53 user init: idmapd main process ended, respawning
Feb  5 11:49:53 user rpc.statd[769]: Caught signal 15, un-registering and exiting.
Feb  5 11:49:53 user init: statd main process ended, respawning
Feb  5 11:49:53 user statd-pre-start: local-filesystems started
Feb  5 11:49:53 user sm-notify[3790]: Already notifying clients; Exiting!
Feb  5 11:49:53 user rpc.statd[3806]: Version 1.1.6 Starting
Feb  5 11:49:53 user rpc.statd[3806]: Flags: 
...

Now, the thing is this - after this bug happens, the server's ssh server is (for some reason) usually still "live", so I can log in via ssh again, and try to close processes (and realize it is impossible to kill /usr/sbin/rpc.nfsd 8, which is the one hanging).

BUT - if at this point, I try to issue a reboot via sudo shutdown -r now && exit from ssh, then this server PC will start the reboot process - but will not complete it; it will drop to a terminal, dump some error messages, and stay there :(

The problem is - the server PC is in a really difficult to access location, and having to go there to do Alt+SysRq + REISUB to properly reboot (if the kernel reacts to that key combo; else it's hard powerdown) is really difficult.

So my question is - is there some "hardcore reboot" command in Linux, that will more-less "guarantee" that the PC will reboot (and not just hang/freeze), even if it has encountered a kernel bug - and which I could issue via ssh? Something that would be the equivalent of a hard powerdown (i.e. turning of the power by e.g. holding the power button for 10+ seconds) and hard powerup?

2 Answers 2

16

To ensure that the system will reboot no matter what, I always do this sequence:

# echo s > /proc/sysrq-trigger
# echo u > /proc/sysrq-trigger
# echo s > /proc/sysrq-trigger
# echo b > /proc/sysrq-trigger

This requests the kernel to do:

  • emergency sync of the block devices
  • mount readonly of all filesystems
  • again a sync
  • force an immediate boot; you can also use o for poweroff.

See e.g. here for explanation of this feature.

5
  • Thanks for that, @wurtel - I tried from my ssh shell: sudo bash -c "( echo a ; sleep 5; echo s > /proc/sysrq-trigger ; echo u > /proc/sysrq-trigger ; echo s > /proc/sysrq-trigger ; echo b > /proc/sysrq-trigger )" & exit; however, this doesn't seem to force the server into reboot... Is there a specific way I should format these commands, if I want to run them while I'm connected through a remote ssh shell?
    – sdaau
    Feb 5, 2015 at 12:10
  • 2
    @sdaau it's possible you first need to enable sysrq. Begin the sequence with echo 1 > /proc/sys/kernel/sysrq
    – Gert
    Feb 5, 2015 at 12:25
  • Thanks, @Gert; I checked with cat /proc/sys/kernel/sysrq and it was already 1, so that was already active, I guess... I think this has something to do with the fact that I'm running this via ssh remote shell, but cannot tell what
    – sdaau
    Feb 5, 2015 at 12:28
  • Some distro's have kernels where magic sysrq is not configured, perhaps that is your problem :-( I have encountered this myself, but I didn't check what the value of proc/sys/kernel/sysrq was so I can't confirm that in this case this is your problem.
    – wurtel
    Feb 5, 2015 at 14:20
  • I can confirm that this works: I had a Raspberry Pi which had gone bad to the point where reboot processes would get suspended in the D state. echo b > /proc/sysrq-trigger did the trick. Aug 16, 2018 at 11:02
1

You have to bypass the normal shutdown process that unmounts filesystems, stops daemons and so on. This is where it stops - it cannot safely stop the processes. What you need is reboot -f or poweroff -f whatever you want to achieve (some init systems may bring their own commands -- systemd for instance). The "force" feature skips the regular shutdown processes and goes directly for hardware reboot.

4
  • Many thanks for that, @orion - I tried from ssh in the controlling terminal this: sudo bash -c "( echo a; sleep 5; reboot -f )" & exit but nothing happens on the server PC... I guess if I issue it manually on the server PC itself it would work, but here I need to issue it from a remote ssh shell, and immediately exit from the shell before the reboot takes place (and the at command doesn't recognize seconds.). Cheers!
    – sdaau
    Feb 5, 2015 at 12:06
  • 1
    To avoid killing the process prematurely, either background the process, call disown to detach the process, and then logout. Or prefix the line with nohup. I prefer the first option.
    – orion
    Feb 5, 2015 at 12:08
  • 1
    Just a note: I just tested several times with sudo reboot -f from ssh (both typed manually and in a script), and realized late that actually there are several instances of reboot -f shown in ps axf with status D which is uninterruptible sleep (usually IO); probably the kernel bug freezes the reboot commands as well...
    – sdaau
    Feb 5, 2015 at 14:23
  • Oh dear. If reboot itself gets stuck in kernel mode (D status means something went wrong in syscall) then I don't see much hope - raw sysrq has a chance, but I'm not sure. And connecting through ssh has nothing to do with anything (it doesn't make a difference)
    – orion
    Feb 5, 2015 at 16:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .