How to change NTP Server settings for Guest Clusters(VKC) on Supervisor when NSX-T is backend network

Products

• VMware vSphere Kubernetes Service
• vSphere with Tanzu

Issue/Introduction

• NTP changes from vCenter UI is not auto reflected in VKCs
• NTP setting Changed through vCenter UI -> Supervisor -> Config -> Network -> Management Network -> NTP but it didn't trigger TKC rolling with vSphere 8

Environment

• VMware vSphere with Tanzu 
• VMware NSX-T

Cause

When NSX-T is integrated with 'vSphere with Tanzu',  NTP update through vCenter UI will only change NTP settings for Supervisor cluster

Resolution

Use below steps from the vCenter's shell command line:

- Identify the cluster name - domain-c10 in below case. 

# /usr/lib/vmware-wcp/decryptK8Pwd.py
Read key from file
Connected to PSQL

Cluster: domain-c10:<cluster UUID>
IP: <IP-address>
PWD: <encrypted-passwd>
------------------------------------------------------------

- Run 'dcli' command to change Workload NTP servers 

# dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c10 --workload-ntp-servers <new-ntp-server>

Once this is done, existing VKCs will start rolling in a few minutes and newly created nodes will have the new NTP servers in /etc/chrony.conf file.

Additional Information

Notes - With TKG service 3.1.0 and older, only the first NTP server will be used inside Workload Clusters even if you set multiple NTP servers. Once vCenter is upgraded to 8.0U3 and TKG service release is upgraded to 3.2.0, customers can create a new cluster using the new ClusterClass shipped in TKG service 3.2.0, which supports multiple NTP servers.