I have a system that uses the
at command to schedule simple commands to be executed later. The command in question uses a lock so that multiple commands started at the same time execute sequentially ("mutex").
At one point I've logged into a machine and saw that something caused ~250 such jobs to be queued. It shouldn't be a serious issue and after waiting for the specified time I saw that commands started running, and it looked sequential - as every command run it was removed from the list returned by
atq, at a rate of about 1 per second.
Eventually though, there were 2 jobs remaining on the queue - well after the time they were supposed to run. Here's the output from
Every 2.0s: atq Mon Jan 4 17:55:58 2016 242 Tue Jan 5 17:44:00 2016 a root 241 Tue Jan 5 17:44:00 2016 a root
$ sudo find /var/spool/cron/at*, I see this:
/var/spool/cron/atjobs /var/spool/cron/atjobs/.SEQ /var/spool/cron/atjobs/a000f201714448 /var/spool/cron/atjobs/a000f101714448 /var/spool/cron/atspool
atd daemon is running, and comparing its PID to other system daemons, it looks like it hasn't crashed and respawned.
I know I can
atrm the jobs and just run the commands by hand, but I'm more interested in understanding why it happens and how to prevent it from happening next time. What could cause these jobs to get stuck?
The system in question is an Ubuntu 12.04 (precise) fully up to date.