NTP Trouble Shooting for VKS Supervisor

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In the Supervisor, inaccurate time synchronization between Supervisor nodes can cause critical issues, including certificate expiration, log inconsistency, and distributed database desynchronization.

This KB describes how to troubleshoot the NTP synchronization on the Supervisor.

# timedatectl status | grep synchronized
System clock synchronized: no

Note: For NTP issues on guest VKS Clusters, please refer to: KB - NTP Troubleshooting for VKS Cluster

Environment

vSphere Kubernetes Service

Resolution

Use the following commands to troubleshoot NTP synchronization on the Supervisor:

SSH to the Supervisor

Check the NTP synchronized status.

timedatectl timesync-status
timedatectl status
timedatectl show-timesync

Check NTP client status.

systemctl status systemd-timesyncd

# Restart the process if it's needed
systemctl restart systemd-timesyncd

# Check the log
journalctl -u systemd-timesyncd --since="120min ago"

Check the current NTP setting.

cat /var/lib/node.cfg | grep NTP
cat /etc/systemd/network/10-eth0.network | grep NTP

Check the network connectivity (UDP:123).

ping -c3 <NTP_SERVER>

# Traceroute verifies the network path to the NTP server, not the actual NTP server response.
traceroute -U -p 123 <NTP_SERVER>

Throw the NTP query to the NTP server.

# To confirm if the NTP service is actually responding
ntpdate -q <NTP_SERVER>
#> server <NTP_SERVER>, stratum 2, offset -0.000278, delay 0.02580
#> DD MM hh:mm:ss ntpdate[#####]: adjust time server <NTP_SERVER> offset -0.000278 sec

# Debug NTP query - Normal state output
sntp -d <NTP_SERVER>
#> Packet sent.
#> sock_cb: <NTP_SERVEER> <NTP_SERVER>:123
#> YYYY-MM-DD hh:mm:ss.sssss (+0000) +0.000008 +/- 0.005605 <NTP_SERVER> s2 no-leap

Additional Information

• KB - How to make temporary changes to the NTP settings with VKS without triggering Guest Cluster Rolling Update
• KB - systemd-timesyncd is flapping in the Supervisors