Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
After a shutdown command is issued, sometimes one gets a status message like this:
A stop job is running for Session 1 of user xy
and then the system hangs for awhile, or forever depending on ???
So what exactly is "a stop job"?
Also, why does it sometimes estimate the time it will take, quite accurately, and other times it can run forever?
systemd operates internally in terms of a queue of "jobs". Each job (simplifying a little bit) is an action to take: stop, check, start, or restart a particular unit.
When (for example) you instruct systemd to start a service unit, it works out a list of stop and start jobs for whatever units (service units, mount units, device units, and so forth) are necessary for achieving that goal, according to unit requirements and dependencies, orders them, according to unit ordering relationships, works out and (if possible) fixes up any self-contradictions, and (if that final step is successful) places them in the queue.
Then it tries to perform the enqueued "jobs".
A stop job is running for Session 1 of user xy
The unit display name here is Session 1 of user xy. This will be (from the display name) a session unit, not a service unit. This is the user-space login session abstraction that is maintained by systemd's logind program and its PAM plugins. It is (in essence and in theory) a grouping of all of the processes that that user is running as a "login session" somewhere.
The job that has been enqueued against it is stop. And it's probably taking a long time because the systemd people have erroneously conflated session hangup with session shutdown. They break the former to get the latter to work, and in response some people alter systemd to break the latter to get the former to work. The systemd people really should recognize that they are two different things.
In your login session, you have something that ignores SIGTERM or that takes a long time to terminate once it has seen SIGTERM. Ironically, the former is the long-standing behaviour of some job-control shells. The correct way to terminate login session leaders when they are these particular job-control shells is to tell them that the session has been hung up, whereupon they terminate all of their jobs (a different kind of job to the internal systemd job) and then terminate themselves.
What's actually happening is that systemd is waiting the unit's stop timeout until it resorts to SIGKILL. This timeout is configurable per unit, of course, and can be set to never time out. Hence why one can potentially see different behaviours.
journalctl to find the process.
Apr 26, 2021 at 19:46
These messages are from systemd, which is a init system which starts and stops jobs. Jobs can be daemons, but can also little tasks such as mounting and unmounting disks, deleting /tmp, or saving and restoring screen brightness across boot. systemctl list-units gives you the idea. Systemd uses "unit" and "job" to mean much the same thing.
When a job is being stopped, as with systemctl stop ..., then a question is how long to wait for the job to complete before declaring failure and killing the job's processes with the SIGKILL signal. We really don't want to use SIGKILL unless we have to, as it doesn't give the opportunity for the process to exit cleanly. For some processes a few seconds might be ample time to declare failure, for other processes like a database there might be substantial network and disk I/O for the job to stop cleanly, and therefore we might give those units several minutes to shut down cleanly.
What you are seeing upon shutdown is the equivalent of systemctl stop $UNIT_NAME which is taking some time to run. There is a counter which shows elapsed seconds and the maximum waiting time before SIGKILL will be issued and the shutdown proceed regardless.
Unless there are good reasons to expect a long delay, this usually indicates some sort of malfunction. That might range from a DHCP server not responding to a Release and thus the Release action needing to time out, or some error causing a daemon to never exit.
Some service is stuck and systemd is waiting for it to exit. Systemd is probably not estimating accurately the time it will take, the time (typically 90 seconds) is how long systemd will wait before it runs out of patience. See this post:
"Stop jobs" are when systemd is waiting for a specific "job" to stop, e.g. some process that it is waiting to complete before continuing onward. If you see a warning message that "a stop job is running..." (etc) it technically means that something is pending in the job queue.
However, before digging through your entire system job queue, keep in mind that sometimes these warning messages are an indirect result from environmental factors (in fact, the message is even referenced on their GitHub repo as a possible bug).
For example: we were getting "stop job" related messages and couldn't figure out why.... turns out, the disk was almost out of space, and it started making the OS behave strangely.
Upgrading the server to a bigger disk and rebooting fixed it ;)
