TKGS Cluster stuck in upgrading state with error spec.kubeadmConfigSpec.mounts: Forbidden: cannot be modified

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

TKGS Cluster stuck in upgrading state with error spec.kubeadmConfigSpec.mounts: Forbidden: cannot be modified

Specific log messaging will present from the vmware-system-tkg-controller-manager pod on the Supervisor Cluster:

2022-08-19T13:11:42.204954642Z stderr F E0819 13:11:42.203550       1 tanzukubernetescluster_controller.go:418] vmware-system-tkg-controller-manager/tanzukubernetescluster-spec-controller/test/test-cluster "msg"="Unable to reconcile control plane for cluster" "error"="Unable to sync KubeadmControlPlane for cluster \"test-cluster\": Failed to update KubeadmControlPlane \"test-cluster-control-plane\": admission webhook \"validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io\" denied the request: KubeadmControlPlane.controlplane.cluster.x-k8s.io \"test-cluster-control-plane\" is invalid: spec.kubeadmConfigSpec.mounts: Forbidden: cannot be modified"  "cluster"="test-cluster"
 

Environment

VMware vSphere 7.0 with Tanzu

Cause

The mounts field in the Control Plane KCP is immutable, causing the TKG controller manager to report errors when reconciling Guest Cluster Control Plane nodes.  

The mounts field was added to allow TKC Nodes to be more resilient to power outages and infrastructure disruptions.

This condition may present when the following actions are attempted:
 

1. Precondition: vSphere with Tanzu TKGS Guest Clusters are running pre-7.0U3 using TKC Volumes (ephemeral mounts) on Control Plane nodes. 
2. The vCenter Server is updated to 7.0U3+.
3. The Workload Management (Supervisor Cluster) is upgraded to a newer version.

• The Supervisor Cluster triggers an automatic upgrade of TKGS Guest Clusters built with the precondition noted in item #1.
• OR; the TKGS Guest Clusters built with the precondition noted in item #1 are manually upgraded to a newer version

 

Resolution

Please contact Broadcom support for assistance with resolving this issue.