Workload Clusters Stuck During Upgrade Due to Orphan AddonReleases in VKS

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

This article describes the manual cleanup required when a workload cluster upgrade becomes stuck due to orphaned Antrea or Calico AddonRelease and AddonConfigDefinition (ACD) resources left behind after a KubernetesRelease (KR) is deleted.

This issue presents intermittently. Because AddonRelease selection becomes non-deterministic when multiple stale releases exist, subsequent upgrade attempts against the identical cluster may exhibit varying results, with some attempts succeeding and others failing.

When the failure occurs, the following symptoms are observed:

• The cluster upgrade hangs and does not progress. Control plane nodes successfully upgrade to the target KR version, but worker nodes remain on the older version.

• The output of kubectl get cluster -A displays AVAILABLE: False for the affected cluster.

• The TopologyReconciled condition is False.

• The AddonsReconciled condition displays Unknown / ReconcileFailed for the CNI ClusterAddon.

• Stale AddonRelease or AddonConfigDefinition objects are present in vmware-system-vks-public for KubernetesRelease versions that no longer exist.

• The runtime-extension controller logs contain:

   

  "addonrelease on clusteraddon has not been updated yet"
    hook="after-control-plane-upgrade" handler="addon-upgrade"
  "addonRelease has not been selected after cluster upgrade was triggered"
    hook="after-control-plane-upgrade" handler="addon-upgrade"
  

Environment

• vSphere Kubernetes Service

• VKS versions prior to 3.6.2 (Environments where a KubernetesRelease add/remove operation was performed and AddonRelease/ACD objects were created without ownerReferences).

• Environments upgraded to 3.6.2+ that still carry legacy orphan resources accumulated prior to the upgrade.

• Cluster upgrades between 1.35.x patch releases (e.g., v1.35.0+vmware.2-vkr.4 to v1.35.2+vmware.1-vkr.3) when a prior 1.35.x KR was deleted without AddonRelease cleanup.

Cause

In VKS versions prior to 3.6.2, Antrea and Calico AddonRelease and ACD objects were created without an ownership link to their parent KubernetesRelease. When such a KR is deleted, these resources are not automatically cleaned up and remain indefinitely.

When a cluster upgrade is triggered, the addon-upgrade handler attempts to select the correct CNI AddonRelease for the target KR version. If multiple stale AddonReleases exist for the same addon version, the selection becomes non-deterministic. The handler may select the orphaned release instead of the one matching the target KR, causing a deadlock.

Note: Do not attempt to manually delete the AddonRelease if the cluster is already stuck, as it is still in use and deletion will be blocked by a validating webhook.

Resolution

The permanent resolution is to upgrade VKS to 3.6.2 or newer, which adds ownerReferences to AddonRelease and ACD objects for automatic garbage collection. If orphan resources accumulated before the upgrade, the following manual cleanup must be applied.

Prerequisites:

• kubectl configured with access to the supervisor cluster.

• Administrator privileges on the supervisor (permissions to get and delete addonrelease and addonconfigdefinition resources).

• jq installed.

Step 1: Download the Cleanup Script The cleanup-orphan-addonresources.sh script attached to this Knowledge Base article must be downloaded to the local workstation or jumpbox where kubectl is configured.

Step 2: Identify orphaned resources (Dry Run) A dry run must be executed using the downloaded script to preview the resources targeted for removal before any deletion occurs:

bash cleanup-orphan-addonresources.sh --dry-run

Step 3: Execute the cleanup Once the dry-run output is confirmed, the cleanup script can be executed to remove the orphaned resources:

bash cleanup-orphan-addonresources.sh --execute

The script performs the following actions:

1. Deletes each orphaned AddonRelease not actively referenced by a live ClusterAddon.

2. Deletes the corresponding AddonConfigDefinition (ACD) referenced by each orphaned AddonRelease.

3. Logs all actions to a timestamped log file (orphan-addon-cleanup-<timestamp>.log).

Recovery Workflow:

• If cleanup is performed before an upgrade: Orphaned resources are removed prior to the upgrade. The handler locates only one matching AddonRelease and the upgrade proceeds successfully.

• If the cluster is already stuck: The cleanup script will skip the orphaned AddonRelease as it is currently in use. KB 445156  must be applied first to redirect the cluster to the correct CNI version and resume the upgrade. Once the upgrade completes, the cleanup script must be re-run to remove the newly freed orphan.

The upgrade is considered complete when:

• The orphan query returns no results (or only results for releases actively in use).

• The target cluster status returns to Healthy / Ready.

• The Topology Reconciled condition transitions to True.

• Cluster machines complete their rolling upgrade to the target KR version.

Additional Information

If the cluster upgrade is actively stuck, the workaround in KB 445156 must be applied first.