NTP Trouble Shooting for VKS Cluster
search cancel

NTP Trouble Shooting for VKS Cluster

book

Article ID: 423691

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

In a VKS Cluster, inaccurate time synchronization between Kubernetes (k8s) nodes can cause critical issues, including certificate expiration, log inconsistency, and distributed database desynchronization.

This KB describes how to verify and resolve NTP time drift on VKS k8s nodes.

Note: NTP issues on the VKS Supervisor are out of scope

Run the following command on the target node to check the synchronization status:

timedatectl status
#> System clock synchronized: no # <--- NTP synchronization has failed.

Environment

vSphere Kubernetes Service

Resolution

If time synchronization is failing, follow these steps to troubleshoot and resolve the issue.

1. Verify NTP Client Configuration

Check the configured NTP servers and the current communication status.

# Check configured NTP servers
grep server /etc/chrony.conf

# Check synchronization status with NTP servers
chronyc sources -v
#> ....
#> MS Name/IP address         Stratum Poll Reach LastRx Last sample
#> ===============================================================================
#> ^ <NTP_SERVER>                   2   6     0    58    +12us[  +18us] +/- 2085us

# Detailed tracking information
chronyc tracking

Example Analysis:

  • Poll column is "6": The polling interval is 2^6 = 64 seconds
  • Reach column is "0": All 8 recent polling attempts have failed

 

2. Check Service Status and Restart

Verify if the chrony service is active.

systemctl status chronyd
systemctl restart chronyd

# If the service fails to start, check the logs for errors
journalctl -r -u chronyd

 

3. Troubleshoot UDP Network Connectivity

Verify if NTP packets (UDP 123) are being sent and received using tcpdump. Monitor for at least the duration of one Polling interval.

tcpdump port 123
#> HH:MM:SS.ss IP <THIS_NODE>.xxx > <NTP_SERVER>.ntp: NTPv4, Client, length 48 # <-- Sending packet
#> HH:MM:SS.ss IP <NTP_SERVER>.ntp > <THIS_NODE>.xxx: NTPv4, Server, length 48 # <-- Return packet

 

Case 1: Only "Sending" packets are observed

The node is sending requests, but the NTP server is not responding. This usually indicates a network issue.

  • Review Firewall rule
  • If using NSX, check the Distributed Firewall and Edge policy

Use the following network command.
Note: Results may be inconclusive if ICMP is prohibited in your environment

# Check L3 network reachability
ping -c3 <NTP_SERVER>

# Check UDP conneciton
traceroute -U -p123 <NTP_SERVER>

Confirming UDP connectivity from the client side alone is unreliable. Definitive diagnosis of network issues requires packet captures on the NTP server.

 

Case 2: No packets observed

If no "Sending" packets are observed even after the polling interval, the issue is likely rooted in chronyd itself.

journalctl -r -u chronyd

 

Case 3: Both "Sending" and "Return" packets are observed

Network connectivity is healthy.  Synchronization may be rejected due to low accuracy (Stratum) of the server or a time offset that is too large for automatic adjustment.

journalctl -r -u chronyd

 

(Option) Emergency Manual Adjustment

If an immediate fix is required, set the time manually via SSH.

systemctl stop chronyd
date -s "2026-01-28 01:11:01"

Additional Information

Powered by Wolken Software