vsphere-csi zone and region labels not applied to new worker node for standard workload cluster while upgrading Kubernetes 1.24.10

Products

• VMware Telco Cloud Automation

Issue/Introduction

When upgrading standard workload cluster of Kubernetes 1.24.10 with vsphere-csi addon which has zone and region configured, the new worker node may not have the zone and region label. vsphere-csi addon status is stuck with Configuring.

Environment

2.3

3.1

Cause

When the control plane of standard cluster is upgraded, it generates the default TKG addon secret for vsphere-csi and vsphere-cpi which have zone and region set to null. Since addons are automatically upgraded after control plane is upgraded, vsphere-csi addon should add zone and region config to TKG addon secret. Sometimes, vsphere-csi addon upgrade may fail with error:

failed to create cluster accessor: failed to get lock for cluster: cluster is locked already

This is normally a transient error when control plane, node pools and addons are upgrading at the same time. If node pool upgrade depends on zone and region config from vsphere-csi addon, it may be stuck forever.

Resolution

Follow this process to upgrade standard workload cluster of Kubernetes 1.24.10:

• Upgrade control plane.
• Wait until vsphere-csi addon is Provisioned and TKG addon secret for vsphere-csi has zone and region in it.
  kubectl get secret -n <workload cluster name> <workload cluster name>-vsphere-csi-addon -o "jsonpath={@.data.values\.yaml}"|base64 -d
  Example: 
kubectl get secret -n lc1 lc1-vsphere-csi-addon -o "jsonpath={@.data.values\.yaml}"|base64 -d
• Upgrade node pools.

Additional Information

Standard workload cluster of Kubernetes 1.24.10 can be created in TCA 2.3, or in TCA 3.1 and 3.1.1 which supports Multi-TKG.

The issue may happen when upgrading standard workload cluster of other Kubernetes versions but does not affect classy workload clusters.