Workload Cluster Upgrade Blocked by NamespaceDeletionDiscoveryFailure
search cancel

Workload Cluster Upgrade Blocked by NamespaceDeletionDiscoveryFailure

book

Article ID: 436922

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • Attempts to upgrade a TKG or VKS cluster (e.g., from v1.30 to v1.31) are rejected by the Cluster API (CAPI) Validation Webhook. The kubectl edit or kubectl patch cluster resource commands fail with the following error:

    Error from server (Forbidden): admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: upgrade cannot be initiated as cluster's AddonsReconciled condition is not True. Message: Addon Vsphere-Pv-Csi is not ready: kapp: Error: Timed out waiting after 30s for resources: [daemonset/vsphere-csi-node (apps/v1) namespace: vmware-system-csi]. Reason: ReconcileFailed 
  • Core packages such as Antrea, vSphere CSI, or secretgen-controller show a ReconcileFailed status or timeout error.

  • Following an unsupported attempt to delete the secretgen-controller PackageInstall, the resource becomes stuck in a Terminating state with a NamespaceDeletionDiscoveryFailure. Describing the secretgen-controller PackageInstall with the command kubectl describe pkgi secretgen-controller -n example-namespace reports the following error:

Status:       Terminating

Conditions:

  Type                                         Status  LastTransitionTime               Reason                  Message

  ----                                         ------  ------------------               ------                  -------

  NamespaceDeletionDiscoveryFailure            True    [timestamp]  DiscoveryFailed         Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: spdx.softwarecomposition.kubescape.io/v1beta1: stale GroupVersion discovery: spdx.softwarecomposition.kubescape.io/v1beta1

  • Checking APIServices shows a status of False with MissingEndpoints:

kubectl get apiservice -A | grep -i kubescape

v1beta1.spdx.softwarecomposition.kubescape.io   kubescape/storage            False (MissingEndpoints)   ##d

Cause

The failure is caused by an orphaned APIService registration (e.g., from third-party add-ons like Kubescape or Keda). When an add-on is improperly removed, its APIService object remains while its backing pods/endpoints are gone. The Kubernetes Discovery Client deadlocks or times out while querying this unreachable endpoint. Since kapp-controller relies on the Discovery Client to reconcile native system packages, these timeouts prevent the cluster from reaching a stable state required for upgrades.

Resolution

Perform the following steps to remove the orphaned APIService registration and resolve the Discovery Client deadlock:

Prerequisite: Ensure your kubectl context is set to the affected Workload Cluster and not the Supervisor cluster.

  1. Identify the failing APIService (Status is 'False' with 'MissingEndpoints'):

    kubectl get apiservice | grep -i False

  2. Remove the stale Kubescape APIService:

    kubectl delete apiservice v1beta1.spdx.softwarecomposition.kubescape.io

  3. Restart the secretgen-controller pod to refresh its state:

    kubectl delete pod -l app=secretgen-controller -n vmware-system-appplatform-operator-system


  4. Confirm that all TKG system packages have recovered:

    kubectl get packageinstall -A

  5. Once all native packages show Reconcile succeeded, re-initiate the cluster upgrade.