8

I'm investigating a very strange effect on some Beagle Bone Black (BBB) boards. We're seeing occasional jumps of a few months in the system clock which always correlate with systemd-timesyncd updating the system clock. We see 2 to 3 of these a week across a fleet of 2000 devices in diverse locations.

We've spent a lot of time checking SNTP but that appears to be behaving normally.

We've finally come up with a hardware issue with the on-board real time clock that can cause it to randomly jump 131072 seconds (36 hours) due to electronic noise. This doesn't immediately sit right, the reported time jump is quite specific and much less than we've observed, however deeper reading on the issue suggests jumps may be more random and may even go backwards.

My question is... How does linux use a real time clock to maintain the system clock?

I want to know if an error with the real time clock would only present itself in the system clock when a timesync agent (ntpd or systemd-timesyncd) updates. Is there any direct link between the system clock and an RTC or is it only used by an agent?


Note: In the first paragraph I mentioned that we're seeing jumps of a few months in the system clock which always correlate with systemd-timesyncd updating the system clock. By this I mean that the very first syslog message after a time jump is a Time has been changed syslog message:

grep 'Time has been changed' /var/log/syslog
Oct  2 23:53:33 hostname systemd[1]: Time has been changed
Nov 21 00:07:05 hostname systemd[1]: Time has been changed
Nov 21 00:05:17 hostname systemd[1]: Time has been changed
Nov 21 00:03:29 hostname systemd[1]: Time has been changed
Nov 21 00:01:43 hostname systemd[1]: Time has been changed
Oct  3 02:07:20 hostname systemd[1]: Time has been changed
Oct  3 06:37:04 hostname systemd[1]: Time has been changed

To the best of my knowledge the only thing that emits these messages is systemd-timesycnd (See source code). Obviously if anyone else knows of other regular systemd syslog messages messages matching these I'm open to suggestions.

  • 1
    I don't think linux kernel would modify system clock using hardware clock. It must be some userspace process doing that – 炸鱼薯条德里克 Oct 29 '19 at 14:14
  • Alright. Preliminary experiments suggest this is not the cause we were looking for. I'm still open to input. @sourcejedi I've found some other burried log messages suggesting connman is updating the system clock. Looks like systemd may be updating on the instruction of connman. – Philip Couling Oct 29 '19 at 23:26
6

I can respond to some of these points, including the title.

[...] which always correlate with systemd-timesyncd updating the system clock. By this I mean that the very first syslog message after a time jump is a Time has been changed syslog message:

grep 'Time has been changed' /var/log/syslog
Oct  2 23:53:33 hostname systemd[1]: Time has been changed

Actually, this message does not tell you what program caused the time jump. It is just a symptom of the time jump.

It happens when the kernel tells systemd the clock has been changed.[*] systemd responds by writing this message to the system log, and then recalculating when any .timer units will need to be triggered.

The message is printed by the program systemd, not by systemd-timesyncd.

More specifically, the message prefix "systemd[1]:" means it comes from process ID 1. PID 1 is the special "init" process. The systemd project also calls this the "system manager", to distinguish it from instances of systemd which manage user services.


The program called systemd does not change the clock after the system has finished booting.

In the current systemd source tree that you link to, the only program that even reads the RTC / hardware clock / hwclock is timedated, and only if you query it using timedatectl.

As I recall, older versions of the systemd program read the hwclock once at boot time, before running any other program, and set the system clock accordingly. In the latest version, systemd does not do this. There is only some hackery around telling the kernel which time zone is used for the hardware clock. (And avoiding triggering something very specific called "time warp").

In other words, current systemd appears to implicitly assume that something else initializes the system clock. In most cases, this will be the kernel.

Look up the kernel build option "Set system time from RTC on startup and resume" - CONFIG_RTC_HCTOSYS.

For a rounded understanding, notice there is also an option "Set the RTC time based on NTP synchronization" - CONFIG_RTC_SYSTOHC.


[*] System clock changes are detected using a Linux-specific feature. See TFD_TIMER_CANCEL_ON_SET.

  • Thanks you've really put me on the right track with this. Would you mind if I edit in an extra section to this to explain my final findings? ETA 24 hours. It looks like the distributed BBB image is to blame not the hardware. – Philip Couling Oct 30 '19 at 13:21
  • I've made it a separate answer in the end, too much of a change. Still thanks again. – Philip Couling Nov 1 '19 at 14:12
4

Thanks very much to sourcejedi for this answer. This really led me to find the right answer.

Answer to the question

How does Linux use a real time clock to maintain the system clock?

It does so only once, during boot. It will not query the RTC again until the next reboot. This is configurable, but will do so by default on most kernel builds.


I want to know if an error with the real time clock would only present itself in the system clock when a timesync agent (ntpd or systemd-timesyncd) updates.

Unless the system is rebooted, the time in the RTC is unlikely to get into the system clock at all. Some agents like ntpd can be configured to use an RTC as a time source but this is not usually enabled by default. It's inadvisable to enable it unless you know the RTC is a very good time source.


Is there any direct link between the system clock?

It appears the time is copied the other way. The RTC is periodically updated with the system time. As per sourcejedi's answer, this is done by the kernel if CONFIG_RTC_HCTOSYS is set.

This can be tested:

  • Set the RTC

    # hwclock --set --date='18:28'
    
  • Then check the RTC time every few minutes with:

    # hwclock
    

The result of this will be the system time will not change at all, and the RTC will eventually revert to the system time.


The cause of time jumps on the BBB

As sourcejedi pointed out, the messages were not being triggered by systemd-timesyncd. They were being triggered by connman. The evidence was (should be) a spurious log message in /var/log/syslog:

Oct  3 00:10:37 hostname connmand[1040]: ntp: adjust (jump): -27302612.028018 sec
...
Nov 21 00:07:05 hostname systemd[1]: Time has been changed

prior to version 1.37, connman is hard coded to promiscuously poll the default gateway for the time. It does not need to be DHCP configured to do this and if connman's NTP client is enabled (it is by default) then it will do this regardless of any other configuration.

In our case some home routers were actually responding to these NTP requests, but the results were highly unreliable. Particularly where the router was rebooted, it continued to hand out the time without actually knowing the correct time.

For example we know that at least one version of the BT Home Hub 5 will, when rebooted, default to 21st November 2018 and give out this date over NTP. It's own NTP client will then correct the problem but there's a window where it hands out 21st November 2018.

That is, this issue was ultimately caused by our customers rebooting their router and connman just accepting this time.

I'll express my frustration here, it seems the belligerence of some has left this "feature" in connman for far too long. It was reported as a problem as early as 2015. And it's a really well hidden "feature". There are no timeservers configured and no log message to explain what connman is doing or documentation as to why. If your test rigs have no NTP server on the default gateway you'll never see this in testing.


How to Fix

We are looking at two options which both appear to work:

  1. Remove connman completely. It seems the network works just fine without it; we've not yet found the reason for it being there in the first place.

    apt-get remove connman
    
  2. Disable NTP in connman by editing /var/lib/connman to include:

    [global]
    TimeUpdates=manual
    
  • 1
    "The RTC is periodically updated with the system time.". This is done periodically by the kernel, if an ntp client tells it the clock has been synced. – sourcejedi Nov 1 '19 at 16:42
  • @sourcejedi I'm being slow, well spotted. I'll edit to correct – Philip Couling Nov 1 '19 at 16:44
-6

man rtc has a good intro: RTC vs system clock. After explaining that the "wall" clock RTC is running on battery and ticking, while the system time is non-existent (power OFF/hibernate etc), it says:

... So at boot time, and after resuming from a system low power state, the system clock will often be set to the current wall clock time using an RTC.

This has nothing to do with any agents, it is something every user expects from his PC, for decades, be it windows or linux. So yes there is a very direct link from RTC to System Clock.

"occasional jumps of a few months in the system clock"

...it got corrected (so it was totally wrong before)?

...or it was correct, and then suddenly (reboot?) jumped to the past? to the future?

Linux? For a "server" you need NTP. systemd-timesyncd is a typical in between thing, according to the man page: can help you out if you have no RTC at all, and has this "minimalistic" SNTP service.

rtc_cmos rtc_cmos: setting system clock to 2019-10-27T07:38:03 UTC (1572161883)

This happens during kernel boot right before the initrd is started -- if this is wrong (unreliable RTC), then it is up to some NTP service to overrule, and maybe even correct (in case the RTC is reliable, but just a bit slow or fast).


I have no idea how to sum up all these quite fruitless comments...

I flash-checked the BBB-link; 50$ for a mainboard. RTC not working properly, and now all these comments? No info?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.