Define NTP servers in class-based workload cluster

Products

• VMware Tanzu Kubernetes Grid
• VMware Telco Cloud Platform

Issue/Introduction

• TKG nodes deployed in air-gapped infrastructure environments experience clock desynchronization over time.
• Workload cluster recently upgraded from legacy to a class-based cluster.

 

Environment

TKG: 2.5.4

Kubernetes: 1.33.1, 1.32.5, 1.31.9, 1.30.13, 1.29.15, 1.28.15, 1.27.16

TCP: 5.1

Cause

1. The workload cluster's spec.topology.variables.ntpServers topology variable is not defined.
2. Legacy workarounds (manual modifications or older ytt overlays) are unsupported and are overwritten during scaling or lifecycle events in TKG 2.x.

Resolution

Update the workload cluster to use the ntpServers topology variable

1. Prerequisites

   1. Ensure UDP Port 123 is open between the TKG environment and the NTP servers.

   2. Verify that the NTP servers will respond to NTP queries from the node network subnets.

2. Create a backup of the current cluster spec:
   
   kubectl get cluster <workloadClusterName> -n <workloadClusterNamespace> -o yaml > cluster-backup-$(date +%Y%m%d-%H%M%S).yaml

3. Define the workload cluster ntpServers topology variable:
   
   kubectl patch cluster <workloadClusterName> -n <workloadClusterNamespace> -p '{"spec":{"topology":{"variables":[{"name":"ntpServers","value":["#.#.#.#","#.#.#.#"]}]}}}'

   Note: The Cluster API controller will initiate a rolling update to roll out new nodes across control plane and worker groups.

   Note: If a booting node cannot connect to the NTP endpoints via UDP 123, the rolling update will intentionally stall on the first machine replacement to prevent compromising the cluster with non-synchronized nodes.

4.  Confirm the variable was applied:

kubectl get cluster <workloadClusterName> -n <workloadClusterNamespace> -o jsonpath='{.spec.topology.variables}'

or watch the rolling update progress:

kubectl get machines -n <workloadClusterNamespace>

Alternate Method: Add ntpServers Without Overwriting Existing Topology Variables

Note: If post-patch verification reveals that other topology variables are missing from spec.topology.variables, run through the following steps to ensure only the ntpServers variable is updated:

1. Restore the cluster resource from the backup taken in the previous steps

   kubectl apply -f cluster-backup-<timestamp>.yaml

2. Confirm the original spec has been reverted:
   
   kubectl get cluster <workloadClusterName> -n <workloadClusterNamespace> -o jsonpath='{.spec.topology.variables}'

3. Once the cluster's original spec has been successfully reverted, edit the workload cluster spec:

   kubectl edit cluster <workloadClusterName> -n <workloadClusterNamespace>

4. Locate the spec.topology.variables array and add the ntpServers entry alongside the existing variables:

   spec:
     topology:
       variables:
       - name: ntpServers
         value:
         - "#.#.#.#"
         - "#.#.#.#"
       - name: <existingVariable>    # existing variables remain untouched
         value: <existingValue>

5. Save and exit the editor. The Cluster API controller will apply the change and initiate a rolling update as described above.

 

Additional Information

TKG 2.5: Configuration File Variable Reference: vSphere