4

The time on my Linux machine will not stay put. I need to test an issue with our server that only occurs on Saturdays.

  • CentOS 6.5 64-bit
  • I have NTPD off. (service ntpd stop and chkconfig ntpd off)
  • I have the virtual BIOS/ HW clock set to May 2, 2015
  • Changing the time to future or past has the same effect.
  • I set the date on the command line with date --set "02 May 2015 11:00:00"
  • date replies properly and shows the date is May 2 2015
  • Nothing related to time/ntp in cron.d, cron.daily, cron.hourly, etc.

After a few minutes the time will force itself back to the proper date.

What else could be running to continually change the time back? Where is it getting its info!?

  • 5
    And i thought my job was hard, here you are trying to stop time... – datUser Apr 28 '15 at 20:10
  • Is this a virtual guest? If so, it's possible the clock is actually synchronised to the host, and therefore outside your control. – roaima Apr 28 '15 at 21:42
  • Do you have systemtap installed? It can be used to trace the stime and settimeofday system calls and identify which process is making the call. – Mark Plotnick Apr 28 '15 at 21:56
  • @JohnWHSmith Sorry, I'm using the iPad stack exchange app, and it didn't show your answer until just now! Your answer is nice because it's more portable - no packages to install. – Mark Plotnick Apr 28 '15 at 22:04
  • @MarkPlotnick And I... deleted my comment too early (I said SystemTap took a lot of fun away from my solution). But let's face it: stap is just another toy I didn't know I had on my system, and I'm definitely going to play with it a lot now... so thanks :D – John WH Smith Apr 28 '15 at 22:20
4

For this answer, I'll assume that there may be several elements working hard to set your time straight. Since I don't really want to wild-guess about which one is working against you, I'll try and give you an answer which should help you find it yourself instead.

On a UNIX system, the clock can typically be set using the stime system call. As things evolved, it also became possible to set the clocks more accurately using the clock_settime call instead. You might also come accross settimeofday. When running date --set on a CentOS machine, strace revealed that it used clock_settime.

Knowing this, a solution would be to trace these system calls. Good thing is, Linux has a mechanism for that: debugfs. On my system, calling mount, I can see that this is available at /sys/kernel/debug :

$ mount
none on /sys/kernel/debug type debugfs (rw)
...

However, on some systems (including RedHat and probably CentOS), it isn't mounted at boot time. You'll therefore need to run...

# mount -t debugfs nodev /sys/kernel/debug

Also note that if you were in that directory before mounting, you might have to go out and back in before files start to appear in it.

Now we're ready to go. Let's enable the trace for our systems calls. I'm tracing all of them because I don't really want to check which one is really being used. Tracing for system calls can be set in /sys/kernel/debug/tracing/events/syscalls. In this directory, you should find...

  • sys_enter_stime
  • sys_enter_clock_settime
  • sys_enter_settimeofday

... depending on what's available on your system.

These correspond to the events of entering our system calls, which is what we want to trace (you'll also find sys_exit_* directories). Within each directory, you'll find a file named enable, the contents of which should appear to be 0. To trace these calls, set that to 1 instead:

# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_stime/enable
# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_clock_settime/enable
# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_settimeofday/enable

Now that we've set up our trap, just wait until something sets your time to its correct value. Once it has happened, run for the trace logs at...

# cat /sys/kernel/debug/tracing/trace 

Now, unless something wrong occured, you should see one of the following lines:

stime-xxxxx [xxx] .... x.x: sys_stime(...)
clock_settime-xxxxx [xxx] .... x.x: sys_clock_settime(...)
settimeofday-xxxxx [xxx] .... x.x: sys_settimeofday(...)

The number right after stime- (or another call's name) is the PID of the process which made the system call. Now go get it:

# ps -fp xxxxx
UID        PID  PPID  C STIME TTY          TIME CMD
root     XXXXX XXXXX  0 hh:mm ?        hh:mm:ss time_warrior

You should now have everything you need to make sure your system stops getting the time right. The simplest thing would probably be to kill the process, and make sure it isn't spawned at boot time ; of course, you'll have to make sure it doesn't serve a more important purpose before doing so : you don't want to completely crash your system...

Also remember to disable the trace when you're done by writing 0 to the files we edited earlier. A shortcut could be:

# echo 0 > /sys/kernel/debug/tracing/events/syscalls/enable

(this file acts as a master switch for all others ; it allows you to switch all system calls tracing off)


Note: as Mark Plotnick said in a comment systemtap could be a slightly easier way to achieve similar results. I'll let him write a stap answer if he feels like it, since I'm not fluent with stap scripts at all.

  • Thank you for the extremely helpful and in depth troubleshooting help. I was able to determine vmtoolsd (vmware tools) was running and caused a time check which pulls info from vcenter server. – Justin Edmands Apr 29 '15 at 19:51
1

If you have systemtap installed, which is likely the case on a CentOS system, it's easy to trace any system call system-wide.

# cat clock.stp
probe nd_syscall.stime, nd_syscall.settimeofday, nd_syscall.clock_settime {
    printf("process %d (%s) called %s(%s) at %d\n", pid(), execname(), name, argstr,
           gettimeofday_us());
}

# stap clock.stp

To test this, in another window run

# date 04301056
Thu Apr 30 10:56:00 EDT 2015

and stap's output will be

process 7520 (date) called clock_settime(CLOCK_REALTIME, [1430405760.000000000]) at 1430405782997282

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.