Cluster ready state shows False due to Rancher and Cattle webhook errors

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

TKr or VKS clusters may be in a not ready state or provisioning failed. 
All pods on the cluster will be Running or Completed successfully. 
When describing the cluster, the following errors are present on the cluster, however rancher is removed or uninstalled from the environment:

                           * failed to configure CPI for run.tanzu.vmware.com/v1alpha3, Kind=TanzuKubernetesCluster <cluster name>: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": service "rancher-webhook" not found
              * failed to configure CSI for run.tanzu.vmware.com/v1alpha3, Kind=TanzuKubernetesCluster <cluster name>: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": service "rancher-webhook" not found
              * failed to configure AuthService for run.tanzu.vmware.com/v1alpha3, Kind=TanzuKubernetesCluster <cluster name>: unable to create or update authsvc: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": service "rancher-webhook" not found

Environment

All TKr or VKS releases

Cause

This is caused by a failure of the uninstall or rollback of a rancher installation and configuration within a Tanzu Kubernetes workload cluster

Resolution

Please engage Rancher support to assist in cleaning up the remaining Rancher artifacts causing errors in the cluster.