How to troubleshoot NTP connectivity

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

This KB describes how to troubleshoot NTP connectivity issues.

Environment

Product Version: 1.8

Resolution

The NTP servers are configured in Bosh Director Tile ---> Director Config ---> NTP Servers. On TAS and TKGO, chrony is used for the implementation of NTP protocols. The NTP Server IPs are added to /var/vcap/bosh/etc/ntpserver and /etc/chrony/chrony.conf.

Manually run a time synchronization and check if the "refid" has the NTP Server IP address:

/var/vcap/bosh/bin/sync-time &
tail -f /var/vcap/bosh/log/sync-time.out

try: 1, refid: 0.0.0.0, correction: 0.000000000, skew: 0.000
try: 2, refid: 0.0.0.0, correction: 0.000000000, skew: 0.000
try: 3, refid: 0.0.0.0, correction: 0.000000000, skew: 0.000

If the refid is 0.0.0.0, then this confirms that NTP synchronization is not successful. Proceed to the next steps to check connectivity to NTP server:
 

• Check chrony sources:

chronyc sources -v

• Check the MS column:
  • * indicates it is synchronized
  • ? indicates there are connectivity problems to the server
• Check chrony tracking:

chronyc tracking

• The Leap Status should be Normal.
• Check UDP connectivity to NTP:

nc -vu NTP_IP_ADDRESS 123

The above command will only verify a UDP connectivity to NTP server, it will not highlight any connectivity problems with the response. To verify that, you would need to run tcpdump and verify if a response is received.

tcpdump -i eth0 host NTP_IP_ADDRESS -w /tmp/ntp.pcap
systemctl restart chrony 
watch chronyc trackin

Another way to check if there is connectivity with the NTP servers is by running a nmap from the client-server (e.g: opsman VM):

nmap -sU -p 123 ping <NTP SERVER IP>

Here is a breakdown of the responses you might see and what they mean. It's common for nmap to actually capture if the communication is being filtered (blocked) even though the port is open in the destination server. 

If there are no connectivity problems with NTP server but the output of chronyc tracking does not reference the NTP IP, then most likely the clock synchronization failed. By default, chronyc waitsync is set to 10 attempts in /var/vcap/bosh/bin/sync-time.You can run sync-time again or else try different values for waitsync.

Checklist:

• Check chrony sources:

chronyc sources -v

• Check the MS column:
  • * indicates it is synchronised
  • ? indicates there are connectivity problems to the server
• Check chrony tracking:

chronyc tracking

• The Leap Status should be Normal.
• Check UDP connectivity to NTP

nc -vu NTP_IP_ADDRESS 123

The above command will only verify UDP connectivity to NTP server, it will not highlight any connectivity problems with the response. To verify that, you would need to run tcpdump and verify if a response is received.
tcpdump -i eth0 host NTP_IP_ADDRESS -w /tmp/ntp.pcap
systemctl restart chrony 
watch chronyc tracking

If there are no connectivity problems with NTP server, but output of chronyc tracking does not reference the NTP IP, then most likely the clock synchronisation failed. By default chronyc waitsync is set to 10 attempts in /var/vcap/bosh/bin/sync-time.
You can run sync-time again or else try different values for waitsync