55

After a shutdown command is issued, sometimes one gets a status message like this:

A stop job is running for Session 1 of user xy

and then the system hangs for awhile, or forever depending on ???

So what exactly is "a stop job"?

Also, why does it sometimes estimate the time it will take, quite accurately, and other times it can run forever?

2
  • 2
    Maybe it should be stopped job? The session has stopped jobs, which are actually not running, and so don't have an opportunity to respond to termination signals.
    – Kaz
    Sep 19, 2016 at 2:07
  • F9 debug shell garbled by "stop job" message? remove the cylon
    – dotbit
    Dec 10, 2019 at 1:27

4 Answers 4

43

systemd operates internally in terms of a queue of "jobs". Each job (simplifying a little bit) is an action to take: stop, check, start, or restart a particular unit.

When (for example) you instruct systemd to start a service unit, it works out a list of stop and start jobs for whatever units (service units, mount units, device units, and so forth) are necessary for achieving that goal, according to unit requirements and dependencies, orders them, according to unit ordering relationships, works out and (if possible) fixes up any self-contradictions, and (if that final step is successful) places them in the queue.

Then it tries to perform the enqueued "jobs".

A stop job is running for Session 1 of user xy

The unit display name here is Session 1 of user xy. This will be (from the display name) a session unit, not a service unit. This is the user-space login session abstraction that is maintained by systemd's logind program and its PAM plugins. It is (in essence and in theory) a grouping of all of the processes that that user is running as a "login session" somewhere.

The job that has been enqueued against it is stop. And it's probably taking a long time because the systemd people have erroneously conflated session hangup with session shutdown. They break the former to get the latter to work, and in response some people alter systemd to break the latter to get the former to work. The systemd people really should recognize that they are two different things.

In your login session, you have something that ignores SIGTERM or that takes a long time to terminate once it has seen SIGTERM. Ironically, the former is the long-standing behaviour of some job-control shells. The correct way to terminate login session leaders when they are these particular job-control shells is to tell them that the session has been hung up, whereupon they terminate all of their jobs (a different kind of job to the internal systemd job) and then terminate themselves.

What's actually happening is that systemd is waiting the unit's stop timeout until it resorts to SIGKILL. This timeout is configurable per unit, of course, and can be set to never time out. Hence why one can potentially see different behaviours.

Further reading

5
  • 3
    According to this answer, unix.stackexchange.com/a/297318/224025 we can change this time. Would it be safe (or would it do any harm) if I change it to zero seconds? Jul 23, 2017 at 0:38
  • 1
    Actually, the final paragraph of this answer and the user manual that I point you to for further reading already tell you about changing the timeout. A question about what a 0s timeout means and is it safe to employ should be asked as a question per How to Ask because it is a follow-on question to a question of what a "stop job" is and why the timeouts vary. I suspect that it might be a good one.
    – JdeBP
    Jul 23, 2017 at 12:07
  • 1
    Is there a way to output on the console the name of the process that does not terminate after SIGTERM? Mar 10, 2021 at 7:48
  • @MartinMonperrus: See this answer for using journalctl to find the process. Apr 26, 2021 at 19:46
  • Thanks a lot @MatthiasBraun it works this way Apr 29, 2021 at 13:57
2

These messages are from systemd, which is a init system which starts and stops jobs. Jobs can be daemons, but can also little tasks such as mounting and unmounting disks, deleting /tmp, or saving and restoring screen brightness across boot. systemctl list-units gives you the idea. Systemd uses "unit" and "job" to mean much the same thing.

When a job is being stopped, as with systemctl stop ..., then a question is how long to wait for the job to complete before declaring failure and killing the job's processes with the SIGKILL signal. We really don't want to use SIGKILL unless we have to, as it doesn't give the opportunity for the process to exit cleanly. For some processes a few seconds might be ample time to declare failure, for other processes like a database there might be substantial network and disk I/O for the job to stop cleanly, and therefore we might give those units several minutes to shut down cleanly.

What you are seeing upon shutdown is the equivalent of systemctl stop $UNIT_NAME which is taking some time to run. There is a counter which shows elapsed seconds and the maximum waiting time before SIGKILL will be issued and the shutdown proceed regardless.

Unless there are good reasons to expect a long delay, this usually indicates some sort of malfunction. That might range from a DHCP server not responding to a Release and thus the Release action needing to time out, or some error causing a daemon to never exit.

3
  • "Systemd uses "unit" and "job" to mean much the same thing." I don't think that is true: roughly speaking, a "job" is a request to do something to a "unit". See @JdeBP's answer for details.
    – Thomas
    Feb 7, 2017 at 16:13
  • The million dollar question is why doesn’t systemd simply echo the name of the job that is being so slow all the time, so that I may later remember to nuke it from orbit?
    – Steven Lu
    Mar 30, 2021 at 19:47
  • There is no need to pile on more abuse on the systemd project. However.. I want to know what drugs the contributor in question was taking when they made the animated red asterisk indicator which adds nothing of value when they could have spent a fraction of the same effort to print the name of the service being waited on for stopping. So that I can steer clear of them.
    – Steven Lu
    Mar 30, 2021 at 19:51
0

Some service is stuck and systemd is waiting for it to exit. Systemd is probably not estimating accurately the time it will take, the time (typically 90 seconds) is how long systemd will wait before it runs out of patience. See this post:

A stop job is running for Session c2 of user

1
  • 11
    How can I figure out, which service hangs?
    – stollr
    Feb 13, 2017 at 11:38
0

"Stop jobs" are when systemd is waiting for a specific "job" to stop, e.g. some process that it is waiting to complete before continuing onward. If you see a warning message that "a stop job is running..." (etc) it technically means that something is pending in the job queue.

However, before digging through your entire system job queue, keep in mind that sometimes these warning messages are an indirect result from environmental factors (in fact, the message is even referenced on their GitHub repo as a possible bug).

For example: we were getting "stop job" related messages and couldn't figure out why.... turns out, the disk was almost out of space, and it started making the OS behave strangely.

Upgrading the server to a bigger disk and rebooting fixed it ;)

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .