17

Usually when I shut down my machine (with the shutdown command) it works pretty quickly, but every once in a while, usually after a borked update or when I break some service's config file I end up with

A stop job is running for Some Broken Service (20min 58s / no limit)

parading across my screen until I give in and hold down the power button until the machine turns off. Sometimes there are multiple jobs, with one actually hung and the others waiting on the hung one, with a 1:30 timeout per job adding up to 20-30 minutes. Sometimes it will eventually turn off after a lot of waiting, sometimes it won't (or I run out of patience).

Is there anything I can do when I discover this to forcibly terminate whatever's hung and allow it to continue shutting down (semi)-gracefully? Mainly, I want to make sure btrfs always has an opportunity to remount-ro and flush-to-disk because I'm running (semi-unstable) raid6. I've heard Ctrl+Alt+Delete should help, but all it seems to do is print ^[[3~ at the bottom of my screen -- is there something I need to do to enable it?

To be clear, I'm not trying to globally turn down or disable this timeout -- my system is usually able to shut down without running in to it -- I just want a way to bypass it interactively when I discover something's wrong. Also, not interested in why this happens or how to solve it -- diagnosing that has to wait until the machine finishes shutting down and starting back up.

The most recent time I ran into this was by writing a udev rule that caused a kernel panic, causing systemd-udev to be permanently hung as far as I can tell. For some reason, that meant docker hung for 30 minutes until the shutdown itself timed out.

It also seems to happen frequently when I have a systemd .mount unit pointing at a NFS share only accessibly over a VPN, when the VPN has died. Stopping the mount unit will then hang forever, causing for some reason my user session and 6-8 other units to hang for 1:30 each (in sequence) until they all time out.

Also used to happen a while back every time I shut down for some inexplicable reason, then was fixed by a kernel upgrade (5.7.something fixed it, if I recall correctly).

Improve this question
0

2 Answers 2

Reset to default
6

Is there anything I can do when I discover this to forcibly terminate whatever's hung and allow it to continue shutting down (semi)-gracefully?

SysRq + REISUB has been 100% safe in my experience. To be honest, I even use a shorter version SysRq + SUB.

Note: Most SysRq functions are disabled by default on most distributions. To re-enable them, create the file /etc/sysctl.d/90-sysrq.conf containing

 kernel.sysrq = 1

This will take effect on next boot, but to enable it immediately run

 sysctl --system
Improve this answer
3
  • Thanks -- will try this next time I run into the issue. Would still prefer something that let other shutdown tasks that are waiting on the hung one to run, but this is certainly better than nothing and should help solve my worries with btrfs.
    – Gavin S. Yancey
    Commented Sep 8, 2020 at 18:51
  • I've never gotten SysRq to work, but looking into it a bit more it looks like that's because it's disabled by default on most distros. I edited in an explanation, but in case it's not accepted -- Put kernel.sysrq = 1 into /etc/sysctl.d/90-sysrq.conf then run sysctl --system to make that take effect immediately.
    – Gavin S. Yancey
    Commented Sep 8, 2020 at 18:56
  • 1
    oh my gosh, that decision to disable the sys request key nearly completely by default (all but reboot! wtf!) has been most annoying numerous time.. what a useless pseudo security gimmick xD
    – eMPee584
    Commented Dec 9, 2020 at 17:35
-3

Whenever I need to shut my system down asap, my go-to is:

init 0

This is the runlevel (out of a total of 7, numbered from 0 to 6) for stopping all system processes. It's never failed me.

Improve this answer
1
  • 3
    However, that wouldn't help in case you shut down the system regularly, and then the "A stop job is running" message appears during shutdown when no virtual consoles are available anymore.
    – AdminBee
    Commented Jul 2, 2021 at 7:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .