I'm investigating a system failure related to NTP and time sync operations. The problem is hard to duplicate, so I went about just doing some sanity checking to see if different conditions would yield expected results.
My question is about the peer status reporting that ntpq uses, and whether what I am seeing under some conditions is reasonable. As I will explain further on, I am having difficulty understanding how ntpq peer reporting can be correct.
I'm asking here on the Unix/Linux site because I am interested in what normal behavior should be, and I'm assuming NTP has a Unix provenance. But for completeness, this ntpq and ntp daemon are running on Windows Server (third party SW from Meinberg, if that is important).
The working hypothesis has been that the NTP server is failing to respond in some way, leading to a fail-over condition (switching to another source) that may be flawed. The steps below were meant to see if we could force a similar failure.
(By the way, in the following explanation, connecting and disconnecting is done simply be connecting or disconnecting the Ethernet cable at the corresponding server.)
With one NTP server connected, one NTP server disconnected, plus a local internal fail-over time base source, running ntpq gave the following results, where the peer reporting looks like this (my notes are included:
* 192.168.a.b *this is the connected time server* 192.168.c.d *this is the disconnected time server* 127.127.1.0
Then the selected peer (indicated by the asterisk) is disconnected and after a few minutes ntpq reports
192.168.a.b *this has been disconnected* 192.168.c.d *this is still not yet connected* * 127.127.1.0 *this is what I would expect*
The first NTP server is reconnected, and ntpq reports the original status
* 192.168.a.b *this has been reconnected* 192.168.c.d *this is still not connected* 127.127.1.0
Then the second NTP server is connected. After a few minutes (3-4), ntpq reports
x 192.168.a.b *this is still connected* x 192.168.c.d *this is now connected for the first time in this test* 127.127.1.0
Note that we had never noticed the 'x' before, but according to the ntp programming manual, it means that server is a 'falsetick': "The peer is discarded by the intersection algorithm as a falseticker".
Questions: Why is the original selected peer not selected? Which timing source is being used at this point?
Then the first server was disconnected.
At that point in time, ntpq would not run, and we learned that ntpd had died. So we restarted ntp services and continued.
Now, with ntpq and ntpd running again, and after waiting a few minutes, ntpq reports
* 192.168.a.b *but this is not connected!* + 192.168.c.d *this is still not connected* 127.127.1.0
The disconnected NTP server has been declared the selected peer! The second NTP server, which is running and connected, is reported to be a candidate: "The peer is a survivor and a candidate for the combining algorithm."
Questions: Why would the disconnected NTP server be declared to be the selected peer? Which timing source is being used at this point?
While this sequence of events was only done to see if we could force the original failure, it didn't. But it did produce some unexpected results that might point to something contributing to the original problem. It's also interesting that, up to this point, the system continues to operate uninterrupted, not reporting any ntp related problem, and not showing any of the other symptoms of the original failure.
The first server is a commercially available GPS driven time server. The second server is running on a VM on it's own independent hardware ( and I don't have other details at the moment.)