Issue with NTP sync on Tanzu control plane node

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

NTP sync with external server was failing for a control plane node for a particular tenant while NTP sync for control plane nodes for other tenants were successful.

Due to this issue the Prometheus metrics were not able to function properly.

Environment

2.5.x

Cause

NTP sync was failing for the control plane nodes since drop rules were configured on the external firewall to block NTP and DNS packets for that particular tenant.

Resolution

Delete or modify the drop rules to accept NTP/DNS packets for the nodes to sync with NTP server.

Additional Information

Troubleshooting steps 

• Check "System clock synchronized" status by login in to control node ssh capv @control-plane-ip and execute "timedatectl status". If the "System clock synchronized " status is "no" continue with troubleshooting
• Check if the UDP port 123 is open with NTP server from the control plane node using command  nc -z -v -u <NTP server IP> 123
• Even if the above command returns successful state, capture NTP packets using tcpdump and check if the packets are flowing in/out of control plane node
• Restart "systemctl restart systemd-timesyncd" and check NTP status 
• Check the NTP server configuration in /etc/systemd/timesyncd.conf.
• If NTP pool is configured with NTP domain name, check if the control plane node is able to resolve the domain name. 
• If possible modify pool with NTP server IP and check if the node is syncing with NTP