Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

On several production platforms we have observed symptoms which appear to suggest that the time of day clock is periodically jumping forward or backward. The jumps are typically around 1 second, typically cancel out (jump forward then backward very shortly thereafter) and happen around 50 times per day. This drift is most noticeable during times of peak application usage, and during periods of high disk I/O operations such as daily backups. These drifts are affecting our soft real-time sensitive application.

Systems are Oracle Netra X4250 and Netra X4270 servers running SLES 11SP2 with 3.0.58-0.6.6-default kernel.

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

We have disabled NTP, but that has not had any effect on the drifts. Are there tools which measure time of day clock drift? How can we avoid this?

These are production platforms, and we cannot recreate the issue in our labs, so my ability to experiment is limited. If left to my own devices, I'll write a tool to measure drift, and perhaps experiment with an HPET clocksource.

share|improve this question
3  
Disabling NTP makes clocks much more unstable... the only reason I can see for NTP not to keep the clock in line is that the clock is out of whack, and NTP refuses to update it (see ntpdate(8) or ntpd(8)). –  vonbrand Mar 7 at 23:39
    
NTPD does track and correct for clock drift, but what you have isn't drift. Drift is consistently in the same direction by roughly the same amount over time. If it randomly jumps forward and backwards, there's no way to predict it, and accommodate for it. –  Patrick Mar 8 at 7:47

4 Answers 4

Are there tools which measure time of day clock drift?

The only tools I'm aware of are the NTP tools which should suffice. You don't have to actually configure ntpd to sync against a given clock source you can just use the -d option to ntpdate to fetch the calculated offset.

Example:

[davisja5@xxxadmvlm08 ~]$ ntpdate -d clock.redhat.com 2>/dev/null | egrep "^offset"
offset -0.004545
[davisja5@xxxadmvlm08 ~]$

-d is the debug option which does the NTP work without actually touching the system clock.

Any advice on how we can avoid this?

I'm not too surprised that you aren't able to reproduce this in dev/test environments since it's probably just due to the hardware clock. If you have hardware support with someone, I would try to get your machines serviced. One possibility is trading out one of the dev machines for this production machine, fixing the former PROD systems and re-introduce it as a dev machine to replace the one that's in PROD now.

Short of that, switching the hardware clock source is about all you can do. If you don't or can't do the swap thing I'd suggest that you do go the hpet route. You can test whether the clock source change messes with system services and then deploy it into production as a hail mary.

share|improve this answer
    
By "measure clock drift", I didn't mean drift from a reference time source, such as NTP gives you. I meant a tool which can detect "jumps" in the time of day clock over a continuous time range. For example, take time of day samplings every 50ms, and report if the difference from the last sampling is too far from 50ms. Such a tool would show if the time of day clock is drifting from the underlying hardware clock for any reason. –  brett Mar 7 at 21:21
1  
Wouldn't the presence of such intervention likely cause more performance degradation than you're hoping to resolve? In all likelihood though, it's a hardware problem, so you're going to need to get the hardware serviced or use a clock source without this issue. tsc is based in the CPU so it makes sense that higher CPU activity would trigger an issue with the hardware clock anyways. If hpet is fast enough for you, then you may just have to try that, get serviced, or do the swap thing. Those are the only options I can see for you. –  Bratchley Mar 7 at 22:39

One solution is to use HPET

See also High Precision Event Timer

To set it as boot parameter use

clocksource=hpet

On older hardware the TSC was often unstable and was disabled by the kernel.

With the advent of multi-core/hyper-threaded CPUs, systems with multiple CPUs, and hibernating operating systems, the TSC cannot be relied on to provide accurate results ...

Wikipedia:Time Stamp Counter

share|improve this answer
    
On a production system exhibiting the clock jitter symptoms I switched the clocksource to hpet. This had no effect on the observed clock jitter symptoms. –  brett Mar 12 at 13:50
    
HPET is external hardware timer and cannot jitter. So this solution seems to be a wrong path. There were a lot of timing issues with older hardware, especially when using virtualization. Did you checked this with different software also? –  user55518 Mar 12 at 14:46

Isn't clock supposed to be monotonous unless someone changes it? Backward jumps shouldn't be possible. There must be something setting the clock - a cron job or some other daemon (for instance a call to hwclock --adjust). I do recall that ntp itself updates statistics for drift and compensates for it routinely and if you fail to run ntp for a long time and get a huge offset, it messes up time for days after it if you don't reset /etc/adjtime. You may have something like that set up - something that readjusts time drift periodically (and causes jumps).

ntp is actually meant to counter this problem.

share|improve this answer
    
That's what I thought as well. My reading of the hardware clock sources suggests that the counter should be monotonically increasing. If that were true, at worst we should observe erratic tick rates, but never jumps back. On a multiprocessor system, I understand that tsc needs to be synchronized amongst the processors - perhaps this is what is causing backwards jumps? –  brett Mar 10 at 16:02

I wrote a more detailed tool to correlate clock measurements with latency symptoms exhibited by our application. This tool seems to rule out what I previously suspected as jitter in the Linux time of day clock.

So long story short, my initial hypothesis was invalid. But I learned a lot about Linux clocks from the answers and links, so thanks to all who responded!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.