vsphere-csi migration namespace issue during upgrades

Products

• VMware Telco Cloud Automation

Issue/Introduction

When Telco Cloud Automation (TCA) 2.3 is upgraded to TCA 3.0, 3.1 or 3.1.1, after management cluster is upgraded to Kubernetes version 1.26, but before upgrading workload cluster with Kubernetes version 1.24, vsphere-csi addon status on some workload cluster may stuck with Configuring.

Environment

3.0, 3.1. 3.1.1

Cause

Between Kubernetes version 1.24 in TKG 2.1.1 and TCA 2.3 and Kubernetes version 1.25 in TKG 2.3 and TCA 3.0, the namespace of vsphere-csi addon changed from kube-system to vmware-system-csi. The version of tca-kubecluster-operator running on management cluster with Kubernetes version 1.26 is expecting vsphere-csi addon in vmware-system-csi namespace on the workload cluster. Thus it can not reconcile vsphere-csi addon successfully on workload cluster still with Kubernetes version 1.24.

Resolution

Resolved in TCA 3.2

Workaround

Follow this process to upgrade workload cluster of Kubernetes 1.24:

• On management cluster , pause the tca-kubecluster-operator.
  • kubectl edit pkgi -n tca-system tca-kubecluster-operator
  • add spec.paused: true
  • kubectl edit deployment -n tca-system tca-kubecluster-operator
  • change spec.replicas from 1 to 0
  • kubectl get po -A|grep kubec

Once workload cluster with Kubernetes version 1.24 is upgraded to version 1.25, vsphere-csi addon status should change to Provisioned.

• On management cluster , resume the tca-kubecluster-operator.
  • kubectl edit pkgi -n tca-system tca-kubecluster-operator
  • remove spec.paused: true
  • kubectl get po -A|grep kubec