1

I've been using a systemd timer reliably for months, but recently got alerted that the triggered service job hadn't run. When I used systemctl list-timers, indeed I saw that no more triggering events were scheduled.

Thinking it was a fluke, I restarted the timer, used list-timers and saw that now the next run was scheduled as expected. But now the timer seems to firing only about once before it stops scheduling new events until the time is restarted.

Thinking this might be a systemd bug, I upgraded the host from Ubuntu 18.04 to Ubuntu 20.04 but the issue persists. What can cause a systemd timer to stop scheduling new events?.

I checked to see if the triggered service was hung and running more than 24 hours, but it's finishing reliably in about 8 hours.

Here's the timer unit:

[Unit]
Description=Runs service
[Timer]
Unit=myservice.service
# Run every day at this time.
OnCalendar=*-*-* 10:00:00


[Install]
WantedBy=timers.target

Here's the .service file, which performs some maintenance tasks that take about 8 hours to complete:

[Unit]
Description=Do things

# Send email if this fails
OnFailure=status-email-devops@%n.service

[Service]
Environment="[email protected]"
User=example
Group=example
UMask=002
ExecStart=do-thing
ExecStartPost=/usr/local/bin/aws cloudwatch put-metric-data --region us-east-1 --namespace Example --dimensions Host=%H --metric-name example --value 1
StandardOutput=journal

This is with systemd v245.

1 Answer 1

2

I found one possible cause: The service run by the timer had developed a bug which turned into an infinite loop. Since systemd timer's normally don't standard a second instance of a service if there's one already running, the timer appeared to have stopped scheduling new events.

One clue I got was comparing a stuck timer and a freshly started one. I used systemctl show foo.timer on the timer to get more detail about the state and saw this:

 SubState=running

The freshly started timer had instead SubState=waiting.

At that point I did what I should have done in the first place, which is to use systemctl status on the target service which of course showed that it was still running from last time, thus blocking the timer from firing again.

3
  • 1
    Yep, this will happen. *.timer will not trigger a unit that is already running. If your unit is a service, you could consider adding RuntimeMaxSec= to the service to stop it and set it to a failure state if it takes too long (i.e. longer than 23 hours).
    – Stewart
    Commented Sep 14, 2021 at 14:55
  • @Stewart Great tip on RuntimeMaxSec= ! Commented Oct 1, 2021 at 13:30
  • You solved a half year problem for me.
    – deanresin
    Commented Aug 16, 2023 at 3:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .