kubectl edit or kubectl patch cluster resource commands fail with the following error:Error from server (Forbidden): admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: upgrade cannot be initiated as cluster's AddonsReconciled condition is not True. Message: Addon Vsphere-Pv-Csi is not ready: kapp: Error: Timed out waiting after 30s for resources: [daemonset/vsphere-csi-node (apps/v1) namespace: vmware-system-csi]. Reason: ReconcileFailed Status: Terminating
Conditions:
Type Status LastTransitionTime Reason Message
---- ------ ------------------ ------ -------
NamespaceDeletionDiscoveryFailure True [timestamp] DiscoveryFailed Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: spdx.softwarecomposition.kubescape.io/v1beta1: stale GroupVersion discovery: spdx.softwarecomposition.kubescape.io/v1beta1
kubectl get apiservice -A | grep -i kubescape
v1beta1.spdx.softwarecomposition.kubescape.io kubescape/storage False (MissingEndpoints) ##d
The failure is caused by an orphaned APIService registration (e.g., from third-party add-ons like Kubescape or Keda). When an add-on is improperly removed, its APIService object remains while its backing pods/endpoints are gone. The Kubernetes Discovery Client deadlocks or times out while querying this unreachable endpoint. Since kapp-controller relies on the Discovery Client to reconcile native system packages, these timeouts prevent the cluster from reaching a stable state required for upgrades.
Perform the following steps to remove the orphaned APIService registration and resolve the Discovery Client deadlock:
Prerequisite: Ensure your kubectl context is set to the affected Workload Cluster and not the Supervisor cluster.
kubectl get apiservice | grep -i Falsekubectl delete apiservice v1beta1.spdx.softwarecomposition.kubescape.io
kubectl delete pod -l app=secretgen-controller -n vmware-system-appplatform-operator-systemkubectl get packageinstall -AReconcile succeeded, re-initiate the cluster upgrade.