403 Access Denied when upgrading cluster in Tanzu Mission Control - Self Managed

Products

• VMware Tanzu Mission Control - SM
• Tanzu Kubernetes Runtime

Issue/Introduction

A 403 Access Denied error occurred within the Tanzu Mission Control Self-Managed (TMC-SM) interface when attempting to initiate a cluster upgrade

Environment

TMC-SM 1.x

Tanzu Kubernetes Runtime

VMware vSphere Kubernetes Service

Cause

TMC-SM is rejecting the upgrade, but not giving specifics on why the rejection is happening 

Resolution

Please review the logs from the Supervisor cluster to see why the VKS Workload cluster is rejecting the upgrade: 

• kubectl config use-context <tkg-cluster-ns>
• kubectl get pods -A | grep capi
• kubectl describe pod -n <capi-namespace> capi-controller-manager-####-#####
• kubectl logs -n <capi-ns> capi-controller-manager-####-### 

Also this can happen due to PDBs blocking the upgrade, this can be seen when describing the affected VKS Workload cluster.