I have a (physical) box running a stripped down Ubuntu; every now and then (6 times in 3 months), the clock jumps backwards by exactly 300 seconds (+- 0.01 seconds; always exactly 300 seconds). It happens from one minute to the next (I have an external machine polling it once per minute).
The box is running 2.6.26-generic (custom compiled kernel), Ubuntu 9.04 (I know, I'm trying to get it updated, but it's semi-embedded). There is nothing in the logs which indicates what happened, and I have a large selection of pool.ntp.org ntp servers, which correct the problem after a while.
Does anyone know what might cause this?
I also have a number of other boxes running the same kernel (binary identical), and minor variations of the same software, which do not have this problem. I have also swapped out the hardware.
Additional 2 (summary of my individual comments):
- I know 9.04 is out of date, I agree it should be updated, and this decision is out of my control. Because management.
- I have tried a large number of ntp servers, and a small number. It still happens in both cases; if I have a large number of ntp servers, then it fixes itself more quickly.
- I have swapped out the hardware
- I am using the same kernel/operating system on another box (with identical hardware), which is not showing the issue.
- Rebooting has not helped. (this problem has been ongoing for about 6 months)
- Uptime is about 3 months. The box is "always on", running a PBX (asterisk).
- Right now, the hwclock matches the software clock exactly - 0.000000 seconds
- I have not been able to find any cron jobs that reads the hardware clock.
- There is no load-related pattern (though the load is quite low anyway).
- It happens during the day and night.
- It does not happen at regular intervals. Of the ones in the last 3 months, half have happened in the last 9 days.
- This is not "drift" - 99% of the time, it is within a tiny fraction of a second, then from one minute to the next, it jumps EXACTLY 300 seconds, backwards. So, one minute it might say it's 3:07:03, matching my other computer to within 1 microsecond, 60 seconds later, it says 3:04:03.
- I can find nothing in the logs.