NTP time always wrong after reboot. It syncs with ESX even with NTP setup

Question

I've installed NTP and the internal NTP server peer is stratum 2. However, every time I reboot the server, the VM time is synced with the ESX host and is then 6 hours ahead.

I did run ntpdate -s x.x.x.x, and corrected it. Nonetheless, after reboot, it is 6 hours ahead again.

Why doesn't NTP handle it? I've enabled ntp and it starts up during boot, but the time is always the ESX time. I'm on Ubuntu 16.04.

Also timedatectl, doesn't show NTP, but systemd.timesyncd is displayed. systemd.timesyncd is disabled and stopped on the VM.

root@host001:~# timedatectl
                      Local time: Fri 2020-05-08 16:00:59 UTC
                  Universal time: Fri 2020-05-08 16:00:59 UTC
                        RTC time: Fri 2020-05-08 08:57:03
                       Time zone: UTC (UTC, +0000)
       System clock synchronized: no
systemd-timesyncd.service active: no
                 RTC in local TZ: no

Answer 1

Whilst VMWare own white papers recommend running an NTP server in a Linux VM installation, by default, the VM time is synchronized with the time of the hypervisor/host where it resides.

If there is a difference in the time of the hypervisor to the actual time, upon boot or if the ntpd daemon is not running, there will arise time differences in VMs hosted in the affected hypervisors.

vSphere Documentation Center - Configure Time Synchronization Between Guest and Host Operating Systems describing VMWare default behaviour:

After time synchronization occurs, VMware Tools checks once every minute to determine whether the clocks on the guest and host operating systems still match. If not, the clock on the guest operating system is synchronized to match the clock on the host.

If the clock on the guest operating system falls behind the clock on the host, VMware Tools moves the clock on the guest forward to match the clock on the host. If the clock on the guest operating system is ahead of that on the host, VMware Tools causes the clock on the guest to run more slowly until the clocks are synchronized.

Regardless of whether you turn on VMware Tools periodic time synchronization, time synchronization occurs after certain operations:

• When the VMware Tools daemon is started (such as during a reboot or power on operation)

• When resuming a virtual machine from a suspend operation

• After reverting to a snapshot

• After shrinking a disk

If such a difference presents itself between the real time, the hypervisor time and the time of the VM(s), several actions should be carried out:

• correcting the time, timezone/enabling NTP in the VMware host/hypervisor;
• disabling in the VMWare side/vmx file of the VM, the synchronization between the VM/Linux and the hypervisor;

• not having access to the hypervisor, disabling the syncincing of the VM with the hypervisor, at the VM/Linux side, upon boot, using vmtools, for it not to compete all the time, with the NTP daemon setting up/drifting the VM time:

  vmware-toolbox-cmd timesync disable
  

  Doing timesync disable is a must, if you are not able to correct the host/hypervisor time, and even more pressing when those differences are higher.

Quoting again from vSphere Documentation Center - Configure Time Synchronization Between Guest and Host Operating Systems

Native time synchronization software, such as Network Time Protocol (NTP) ..., is typically more accurate than VMware Tools periodic time synchronization and is therefore preferred. Use only one form of periodic time synchronization in your guests. If you are using native time synchronization software, turn off VMware Tools periodic time synchronization.

As for the NTP service on the VM side, ntpd aborts if the time difference is too big, or otherwise syncs it very slowly, if told to ignore it.

For upon boot/NTP service start, the time change to be made immediately and automatically, add as the first line of ntp.conf:

tinker panic 0

See also:

VMWare KB - Disabling Time Synchronization (1189), for disabling completely the time synchronization between the host and the VM.

11 minute mode when more than 30 minutes out of sync for a more detailed explanation of tinker panic 0

Adenda

Nevertheless, spelling it out again, it is strongly advisable to correct the host side notion of time, and bring that up to the attention of the VMWare team, if working with multidisciplinary teams.

Having one hypervisor off-time does affect logs timestamps and VMs/housekeeping files creation/modification times, generated by that VMWare host.

The implications of having hypervisors with the wrong time can be more convoluted, especially if:

• storage space where the VMWare files reside is shared by multiple VMWare hosts;
• logs are being sent to a central syslog server;
• having several VMWare hosts being managed by the same vCenter.

Answer 2

background

On most computers, at booting time:

• The time is read from the RTC (which has been working from a battery while the computer was off).
• Some adjustment might be added depending on what was stored on a file the last time the computer was shut-down.
• That is the new "hardware time", and the time of the computer.
• Until some time client (sntp, ntpdate, ntpd, chronyd, etc), or hardware clock (GPS) updates that time to a better estimate.

If, for any reason, the RTC battery fails, the time at boot could be wildly off the correct time.

Answer

In a virtual computer, the RTC is not a real RTC but a virtual RTC, the time is read from the host computer time. If the host time is off, the boot time of the VM will be also wildly off. The same as if the RTC battery got bad.

That's why this happens (what you copied from the VM):

root@host001:~# timedatectl
                      Local time: Fri 2020-05-08 16:00:59 UTC  
                  Universal time: Fri 2020-05-08 16:00:59 UTC  
                        RTC time: Fri 2020-05-08 08:57:03  
                       Time zone: UTC (UTC, +0000)  
       System clock synchronized: no  
systemd-timesyncd.service active: no  
                 RTC in local TZ: no  

The RTC time is different than real, or local, or UTC.

Solution

Ensure the time of the host is correct. Either use ntp or GPS there.

---

For example, from Qemu man page:

-rtc [base=utc|localtime|datetime][,clock=host|rt|vm][,driftfix=none|slew]

By default the RTC is driven by the host system time. This allows using of the RTC as accurate reference clock inside the guest, specifically if the host time is smoothly following an accurate external reference clock, e.g. via NTP.

Related:Tricking the VM BIOS RTC to set a time