How to check if NTPD updates machine's time successfully using shell?

Question

I'm trying to use NTPD to update my Linux machine's time to a specified NTP server.
Here is the scenario:

Each time the Linux machine starts up, I want to update the time from NTP server and if it's not successful, I want to try again every 5 minutes until successfully (max is 2 hours).

I searched around and find that I should(?) use NTPD and use some command like:

#ntpdate ntp.server.com (before starting NTPD)
#ntpd some_options_to_start

The questions are:

1. How can I know if the time was successfully updated by these commands?
2. Can I set the interval to update time from ntpd? (or I have to use something like sleep and loop with do..while/for in shell?)

Note that I want to execute the above commands in a shell script and will put the shell in a web server. Then clients (with a web browser browser) will execute the script on the website. So I need to check if the update is successful or not to send result to the client (over the web).

Answer 1

Using a script to monitor ntpd is not commonly done. Usually a monitoring tool like nagios or munin is used to monitor the daemon. The tool can send you an alert when things go wrong. I have munin emailing me if the offset exceeds 15 milliseconds.

Normally, you should use an odd number of servers so that the daemon can perform an election among the servers if one goes off. Three is usually adequate, and more than five is excessive. Clients on your internal network should be able to get by with one internal server if you monitor it. Use legitimate servers or your ISPs NTP or DNS servers as clock sources. There are public pools as well as public servers.

ntpd is self tuning and you should not need to adjust it once it is configured and started. With recent ntpd implementations you can drop use of ntpdate entirely as they can do the initial setting of the date.

The following script will parse the offsets in the output of ntpd and report an excessive offset. You could run it from cron to email you if there are problems. The script defaults to alerting on an offset of 0.1 seconds.

#!/bin/bash
limit=100   # Set your limit in milliseconds here
offsets=$(ntpq -nc peers | tail -n +3 | cut -c 62-66 | tr -d '-')
for offset in ${offsets}; do
    if [ ${offset:-0} -ge ${limit:-100} ]; then
        echo "An NTPD offset is excessive - Please investigate"
        exit 1  
    fi  
done
# EOF

Answer 2

To answer the first question, ntpdate usually tells you what it did, or maybe did not do.

[root@flask rc.d]# ntpdate dagoo
12 Aug 10:04:03 ntpdate[20585]: adjust time server 10.0.0.15 offset -0.042285 sec

The NTP daemon, ntpd, runs constantly, and asks NTP servers (usually configured in /etc/ntp.conf) for the time every so often. You shouldn't have to run your script every 5 minutes. ntpdate should bring the machine into near synch with the server, and ntpd will run in the background and keep it in synch. You don't set the interval that ntpd tries, it adjusts the interval based on how it perceives the local clock drifts from the servers, and the quality of the connections to the servers.

You can use a program named ntpdc to see what ntpd keeps as information:

1 % ntpdc 
ntpdc> peers
     remote           local      st poll reach  delay   offset    disp
=======================================================================
*min-time-01.ine 10.0.0.15        1 1024  377 0.07047  0.014673 0.14360
=dns-01.esd189.o 10.0.0.15        2 1024  377 0.07587  0.022277 0.13660
ntpdc>

I think that the number you typically have an interest in is "offset", that's the number of seconds your local clock is off from the server's clock.

As the man page for ntpdc states for the "peers" command :

the current estimated delay, offset and dispersion of the peer, all in seconds.

So, clearly, the "offset" is in seconds.

It appears that ntpdc is deprecated, replaced by ntpq. ntpq has a "peers" interactive command, that does give "offset" in milliseconds. My Redhat server has both ntpdc and ntpq, so you'll need to be careful.

Answer 3

Use ntpstat.

myserver # ntpstat
synchronised to NTP server (10.89.160.13) at stratum 4
   time correct to within 124 ms
   polling server every 1024 s

Answer 4

ntp-wait was made for this problem.

Five minutes with man ntp-wait and you should be up and running...

Answer 5

I added to the @BillTHor bash script also a check for the ntpdstat exit code > 0:

#!/bin/bash
ntpstat > /dev/null
if [ $? -ne 0 ]; then
        echo "NTPD not synchronized - Please investigate"
        exit 1
fi
limit=1000   # Set your limit in milliseconds here
offsets=$(ntpq -nc peers | tail -n +3 | cut -c 62-66 | tr -d '-')
for offset in ${offsets}; do
    if [ ${offset:-0} -ge ${limit:-100} ]; then
        echo "An NTPD offset is excessive - Please investigate"
        exit 1
    fi
done
# EOF

[UPDATE] since the script using ntpq output was unuseful for larget offset (more than 4 digits of offset) I tried a new version using only ntpstat:

#!/bin/bash
ntpstat > /dev/null
if [ $? -gt 0 ]; then
        echo "NTPD not synchronized - Please investigate"
        exit 1
fi
limit=1000   # Set your limit in milliseconds here
#offsets=$(ntpq -nc peers | tail -n +3 | cut -c 62-66 | tr -d '-')
#offsets=$(ntpq -nc peers | tail -n +3 | tr -s ' ' | cut -d ' ' -f 9 | tr -d '-' |tr -d '.')
offsets=$(ntpstat | tail -n +2 | head -n 1 | cut -c 27- | tr -d ' ms')
for offset in ${offsets}; do
    if [ ${offset:-0} -ge ${limit:-100} ]; then
        echo "NTPD offset is excessive: ${offset:-0} [ms] > ${limit:-100} [ms] - Please investigate"
        exit 1
    fi
done
# EOF`enter code here`

Answer 6

Chrony is claimed to handle your use case better than NTPd (network and machine on/off, suspend, etc.). See

http://fedoraproject.org/wiki/Features/ChronyDefaultNTP

RE why do I think chronny is good: It came pre-installed on my fedora machine and I've never had any issue with it (used it for years now). I've never had issues with ntpd in the past as well but if you read on the link I have provided, there is some information as to why chrony is better for non-always on machines. That's why I suggested to the op to try it out, it may or may not work better for him. So it's just another good option to try before going into too much tuning, optimizing and hacking ntpd.

Answer 7

NTP offset can be obtained with the following UNIX pipeline:

/usr/sbin/ntpq -pn | /usr/bin/awk 'BEGIN { offset=1000 } $1 ~ /^\*/ { offset=$9 } END { print offset }'

NTP peer count can be obtained with the following UNIX pipeline:

/usr/sbin/ntpq -pn | egrep -c '^\*|^\+'

For NTP offet we use:

• warning > 250ms
• critical > 500ms

For NTP peer count we use:

• no warning threshold
• critical < 1

Zabbix-ready NTP monitoring config (source: Joyent):

# NTP
UserParameter=ntp.offset,/usr/sbin/ntpq -pn | /usr/bin/awk 'BEGIN { offset=1000 } $1 ~ /^\*/ { offset=$9 } END { print offset }'
UserParameter=ntp.peers,/usr/sbin/ntpq -pn | egrep -c '^\*|^\+'

Nagios-ready NTP monitoring plugins:

check_ntp_offset:

#!/bin/bash
# thresholds
thresh_warn=250
thresh_crit=500

# metric
ntp_offset=$(/usr/sbin/ntpq -pn | /usr/bin/awk 'BEGIN { offset=1000 } $1 ~ /^\*/ { offset=$9 } END { print offset }')

# Exit codes
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

if [[ ! "$ntp_offset" =~ ^[0-9]+$ ]] ; then
   # NTP offset could not be read successfully
   echo "NTP OFFSET UNKNOWN - $ntp_offset"
   exit $STATE_UNKNOWN
elif [[ "$ntp_offset" -gt "$thresh_crit" ]] ; then
   # NTP offset is higher than the critical threshold
   echo "NTP OFFSET CRITICAL - ${ntp_offset}ms (> ${thresh_crit}ms)"
   exit $STATE_CRITICAL
elif [[ "$ntp_offset" -gt "$thresh_warn" ]] ; then
   # NTP offset is higher than the warning threshold
   echo "NTP OFFSET WARNING - ${ntp_offset}ms (> ${thresh_warn}ms)"
   exit $STATE_WARNING
else
   # NTP offset is within thresholds
   echo "NTP OFFSET OK - ${ntp_offset}ms (< ${thresh_warn}ms)"
   exit $STATE_OK
fi

check_ntp_peers:

#!/bin/bash
# thresholds
thresh_warn=1
thresh_crit=1

# metric
ntp_peers=$(/usr/sbin/ntpq -pn | egrep -c '^\*|^\+')

# Exit codes
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

if [[ ! "$ntp_peers" =~ ^[0-9]+$ ]] ; then
   # NTP peers could not be read successfully
   echo "NTP PEERS UNKNOWN - $ntp_peers"
   exit $STATE_UNKNOWN
elif [[ "$ntp_peers" -lt "$thresh_crit" ]] ; then
   # NTP peers is lower than the critical threshold
   echo "NTP PEERS CRITICAL - $ntp_peers (< $thresh_crit)"
   exit $STATE_CRITICAL
elif [[ "$ntp_peers" -lt "$thresh_warn" ]] ; then
   # NTP peers is lower than the warning threshold
   echo "NTP PEERS WARNING - $ntp_peers (< $thresh_warn)"
   exit $STATE_WARNING
else
   # NTP peers is within thresholds
   echo "NTP PEERS OK - $ntp_peers (> $thresh_warn)"
   exit $STATE_OK
fi

I really should let the warning and critical thresholds in the Nagios scripts be configurable with -w and -c. They're not really fully plugin-ready without that. Further guidance on that in a tutorial here: http://www.kernel-panic.it/openbsd/nagios/nagios6.html

Answer 8

#!/bin/bash

limit=100   # Set your limit in milliseconds here

offsets=$(ntpq -nc peers | tail -n +3 | awk '{print $9 }' | tr -d '-')

for offset in ${offsets}; 
do

    if [ ${offset:-0} -ge ${limit:-100} ];
    then
        echo "An NTPD offset is excessive - Please investigate"

        exit 1

    fi  
done

Answer 9

#!/usr/bin/bash
#set -x
NTPLIMIT=100   # Set your NTPLIMIT in milliseconds here
offsets=$(ntpq -nc peers | tail -3 | cut -c 62-66 | tr -d '-')
for offset in ${offsets}; do
    if [ ${offset:-0} -ge ${NTPLIMIT:-100} ]; then
        echo "An NTPd offset is excessive Please investigate" exit;
        else
                echo "NTP is within 0.1 second of time server"
                exit;
        fi
done

Same as the previous answer above, but with a slight modification as the previous command will run the if statement for as many offsets, i.e. if offset is 3 then it will print NTP is within 0.1.... 3 times before closing. Could be annoying if you have a server that is a long way out of sync. There is probably a way to remove the for loop as well...