System clock not synchronized with NTP server using systemd-timesyncd

Question

I followed this answer here, but it seems that my system clock doesn't synchronize with NTP server:

$ cat /etc/debian_version
10.9
$ egrep -v "^$|^#" /etc/systemd/timesyncd.conf
[Time]
NTP=x.y.z.t1
FallbackNTP=x.y.z.t2
$ sudo timedatectl set-ntp true
$ sudo systemctl restart systemd-timesyncd
$ systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Wed 2022-08-24 16:46:29 CEST; 2ms ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 23412 (systemd-timesyn)
   Status: "Idle."
    Tasks: 2 (limit: 4915)
   Memory: 1.4M
   CGroup: /system.slice/systemd-timesyncd.service
           └─23412 /lib/systemd/systemd-timesyncd

Aug 24 16:46:29 EncoderBack systemd[1]: Starting Network Time Synchronization...
Aug 24 16:46:29 EncoderBack systemd[1]: Started Network Time Synchronization.
$ timedatectl timesync-status
       Server: x.y.z.t1 (x.y.z.t1)
Poll interval: 1min 4s (min: 32s; max 34min 8s)
 Packet count: 0
$ timedatectl show-timesync
SystemNTPServers=x.y.z.t1
FallbackNTPServers=x.y.z.t2
ServerName=x.y.z.t1
ServerAddress=x.y.z.t1
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=1min 4s
Frequency=0
$ journalctl -u systemd-timesyncd.service -n 5
-- Logs begin at Mon 2022-08-22 15:20:05 CEST, end at Wed 2022-08-24 16:46:29 CEST. --
Aug 24 16:46:29 EncoderBack systemd[1]: Stopping Network Time Synchronization...
Aug 24 16:46:29 EncoderBack systemd[1]: systemd-timesyncd.service: Succeeded.
Aug 24 16:46:29 EncoderBack systemd[1]: Stopped Network Time Synchronization.
Aug 24 16:46:29 EncoderBack systemd[1]: Starting Network Time Synchronization...
Aug 24 16:46:29 EncoderBack systemd[1]: Started Network Time Synchronization.
$ timedatectl status
               Local time: Wed 2022-08-24 16:46:29 CEST
           Universal time: Wed 2022-08-24 14:46:29 UTC
                 RTC time: Wed 2022-08-24 14:46:19
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
$

EDIT0 : Here is a tcpdump trace while restarting systemd-timesyncd.service :

$ sudo tcpdump -v dst port 123
tcpdump: listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:46:34.136278 IP (tos 0x10, ttl 64, id 18841, offset 0, flags [DF], proto UDP (17), length 76)
    ntpclient.lan.53695 > ntpserver.lan.ntp: NTPv4, length 48
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3870427594.031728329 (2022/08/25 16:46:34)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3870427594.031728329 (2022/08/25 16:46:34)
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel

EDIT1 : Here is a tshark trace while restarting systemd-timesyncd.service :

$ sudo tshark -n -f 'udp port 123' -c2
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eno1'
    1 0.000000000 a.b.c.d  → x.y.z.t1 NTP 90 NTP Version 4, client
    2 0.000678872 x.y.z.t1 → a.b.c.d  NTP 90 NTP Version 3, server
C2 packets captured

EDIT2 : Thanks to @Bib and to the tshark output, it seems the systemd-timesyncd client sends NTPv4 protocol requests but the server responds with NTPv3 protocol answers.

As @QuartzCristal and @Bib suggest, I will be using ntpsec.

EDIT3: After having configured the /etc/ntpsec/ntp.conf file and restarted the ntpsec service, it works fine now :

$ grep ^server /etc/ntpsec/ntp.conf
server x.y.z.t1 iburst
server x.y.z.t2 iburst
$ sudo mkdir /var/log/ntpsec/
$ sudo chown ntpsec:ntpsec /var/log/ntpsec/
$ sudo systemctl restart ntpsec
$ systemctl status ntpsec.service
● ntpsec.service - Network Time Service
   Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2022-08-26 11:06:49 CEST; 2s ago
     Docs: man:ntpd(8)
  Process: 22622 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
 Main PID: 22625 (ntpd)
    Tasks: 1 (limit: 4915)
   Memory: 1.6M
   CGroup: /system.slice/ntpsec.service
           └─22625 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec

Aug 26 11:06:49 EncoderBack ntpd[22625]: CLOCK: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2021-12-28T00:00Z last=2017-01-01T00:00Z ofs=37
Aug 26 11:06:49 EncoderBack ntpd[22625]: CLOCK: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): expired less than 242 days ago
Aug 26 11:06:49 EncoderBack ntpd[22625]: INIT: Using SO_TIMESTAMPNS
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen and drop on 0 v6wildcard [::]:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen and drop on 1 v4wildcard 0.0.0.0:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen normally on 2 lo 127.0.0.1:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen normally on 3 eno1 a.b.c.d:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen normally on 4 lo [::1]:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listen normally on 5 eno1 [fe80::3e7c:3fff:fed4:a223%2]:123
Aug 26 11:06:49 EncoderBack ntpd[22625]: IO: Listening on routing socket on fd #22 for interface updates

Now the system clock is synchronized :

$ timedatectl
               Local time: Fri 2022-08-26 11:08:05 CEST
           Universal time: Fri 2022-08-26 09:08:05 UTC
                 RTC time: Fri 2022-08-26 09:08:05
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no

EDIT4 : Here is a tcpdump output of what is going on when using ntpsec, the source packet tos has changed and the source port is now 123 :

$ sudo tcpdump dst port 123 -n -c 2 -v
tcpdump: listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:53:49.185280 IP (tos 0xb8, ttl 64, id 54505, offset 0, flags [DF], proto UDP (17), length 76)
    a.b.c.d.123 > x.y.z.t1: NTPv4, length 48
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 32
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   1839874488.898661747 (2094/05/28 04:43:04)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 1839874488.898661747 (2094/05/28 04:43:04)
11:53:49.185929 IP (tos 0x0, ttl 126, id 18818, offset 0, flags [none], proto UDP (17), length 76)
    x.y.z.t1.123 > a.b.c.d.123: NTPv3, length 48
        Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 0 (1s), precision -23
        Root Delay: 0.000000, Root dispersion: 10.751129, Reference-ID: LOCL
          Reference Timestamp:  3870431575.277677199 (2022/08/25 17:52:55)
          Originator Timestamp: 1839874488.898661747 (2094/05/28 04:43:04)
          Receive Timestamp:    3870496473.230674199 (2022/08/26 11:54:33)
          Transmit Timestamp:   3870496473.230678499 (2022/08/26 11:54:33)
            Originator - Receive Timestamp:  +2030621984.332012452
            Originator - Transmit Timestamp: +2030621984.332016752
2 packets captured
2 packets received by filter
0 packets dropped by kernel

And here is a tshark output of what is going on when using ntpsec, the weird is that it is the same output as the one I got from using systemd-timesyncd.service (except the source port is now 123) :

$ sudo tshark -f 'udp port 123' -n -c 2
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eno1'
    1 0.000000000 a.b.c.d  → x.y.z.t1 NTP 90 NTP Version 4, client
    2 0.000787978 x.y.z.t1 → a.b.c.d  NTP 90 NTP Version 3, server
2 packets captured

Answer 1

You could install ntpdate or/and sntp and test the server. Both packages are relatively small. Follow the guide in https://weberblog.net/basic-ntp-client-test-ntpdate-sntp/

If the servers work correctly from the computer that should use them, then, you need to test the client. There is no simple way to test the systemd use of ntp. You probably will find a faster solution by removing the systemd-timesyncd.service and install a real ntp package like chrony or ntpsec.

$ sudo systemctl stop systemd-timesyncd.service
$ sudo systemctl disable systemd-timesyncd.service
$ sudo systemctl mask systemd-timesyncd.service
$ sudo apt install ntpsec

Edit the /etc/ntpsec/ntp.conf configuration file and restart the ntpsec service :

$ sudo systemctl restart ntpsec

And then look at the status of the ntpsec server:

$ systemctl status ntpsec.service
● ntpsec.service - Network Time Service
     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
     Active: active (running) since Thu 2022-08-25 19:21:58 AST; 2s ago
       Docs: man:ntpd(8)
    Process: 88959 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
   Main PID: 88962 (ntpd)
      Tasks: 1 (limit: 14032)
     Memory: 10.9M
        CPU: 11ms
     CGroup: /system.slice/ntpsec.service
             └─88962 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec

$ ntpq -pn
     remote                                   refid      st t when poll reach   delay   offset   jitter
=======================================================================================================
 0.debian.pool.ntp.org                   .POOL.          16 p    -  256    0   0.0000   0.0000   0.0001
 1.debian.pool.ntp.org                   .POOL.          16 p    -  256    0   0.0000   0.0000   0.0001
 2.debian.pool.ntp.org                   .POOL.          16 p    -  256    0   0.0000   0.0000   0.0001
 3.debian.pool.ntp.org                   .POOL.          16 p    -  256    0   0.0000   0.0000   0.0001
-45.79.202.46                            216.239.35.0     2 u   22   64    1  65.0429   3.5154   2.1775
+129.250.35.250                          129.250.35.222   2 u   22   64    1  50.2368   1.9173   3.1973
-147.182.158.78                          15.254.136.119   2 u   22   64    1  88.7377   5.4637   1.6545
...
...

Answer 2

An old thread but worth adding my tuppence worth for those that see the same problems as me and waste 6 hours of their lives i.e. timedatectl is showing;

System clock synchronized: no

And the service status shows;

● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Tue 2023-10-03 15:10:48 BST; 4s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 4302 (systemd-timesyn)
   Status: "Idle."
    Tasks: 2 (limit: 2178)
   CGroup: /system.slice/systemd-timesyncd.service
           └─4302 /lib/systemd/systemd-timesyncd

Oct 03 15:10:48 SMARTbox systemd[1]: Starting Network Time Synchronization...
Oct 03 15:10:48 SMARTbox systemd[1]: Started Network Time Synchronization.

I tried absolutely everything until I found how you can run most systemd services with debug output interactively Running systemd Interactively

The service runs fine locally, which then made me ask why isn't it running as a service properly? It looks as though it's waiting for something....

I added the debug setting to the service file and started it and then I got to the root of the problem - the service cannot resolve the NTP names.

I checked the resolv.conf file and that is fine, it all works from outside of systemd. However, I'm running in a near read-only state (to minimise writes to SD card) so the resolv.conf is sym-linked to a tmpfs location.

I reverted the file to being non-symlinked and it now works.