If you have root access on the nodes, then to set up time synchronization. The most commonly used tool for this is ntpd, available in all but the most specialized Linux distributions (as well as other Unix systems). Ntpd periodically sends queries over the network to obtain the current time, measures the drift of the local clock and tunes the local clock to compensate for its skew. You can get better than millisecond precision over a LAN.
For best results, select one or a few nodes as servers, e.g. the head node(s). Install the NTP server package (e.g.
ntp on Debian/Ubuntu/Mint as well as on Fedora/RHEL/CentOS) and configure the servers to be your head node(s) as well as some Internet servers. In the configuration file
/etc/ntp.conf, you should have something like
(leave the lines set up by your distribution and add yours). On the head nodes, if there is a
restrict line, make sure it allows the IP addresses of all the nodes. On the other nodes, you can have
Before you set up NTP, set the clocks to approximately the right time. Ntpd will not synchronize the time if it's more than about 20 minutes off (it assumes that you've set up the timezone incorrectly, or there's something very wrong about the clock or about the network). So first set the clock manually, e.g. first set the correct date and time on the local machine, then you can use
ssh somenode "date -d $(date +%m%d%H%M%Y.%S)"
to set the clock of
somenode to a few tens of a second behind the local clock. Once you've done that, install ntp, or, if it's already installed, restart the daemon with
service ntpd restart or whatever method your distribution uses.