It seems to be a known
systemd bug in Arch. Did not find any reference for Debian itself, but is possible there is a similar bug. From the arch wiki relevant post
systemd-timesyncd will contact servers in turn until one is found that
responds. Arch uses the zones in the order of 0, 1, 2, 3. However, all
pool.ntp.org zones only have IPv6 and IPv4 enabled on the 2 subdomain.
All others are IPv4 only. Therefore, 2 should be placed first in order
to properly provide support for IPv6/IPv4 hosts.
It is also possible DHCP is giving you a wrong NTP server.
The following settings are configured in the "[Time]" section:
A space-separated list of NTP server host names or IP addresses.
During runtime this list is combined with any per-interface NTP
servers acquired from systemd-networkd.service(8). systemd-timesyncd
will contact all configured system or per-interface servers in turn
until one is found that responds. This setting defaults to an empty
A space-separated list of NTP server host names or IP addresses to be
used as the fallback NTP servers. Any per-interface NTP servers
obtained from systemd-networkd.service(8) take precedence over this
setting, as do any servers set via NTP= above. This setting is hence
only used if no other NTP server information is known. If this option
is not given, a compiled-in list of NTP servers is used instead.
Possible workaround: disabling
systemd-timesyncd and installing