Tanzu Hub build-to-build upgrades hang due to stale ReconcileFailed state on unchanged PackageInstall resources

Products

• VMware Tanzu Platform Core
• VMware Tanzu Platform - Hub
• Tanzu Hub SaaS
• VMware Tanzu Platform
• Vmware Tanzu Platform - SM

Issue/Introduction

During build-to-build upgrades of Tanzu Hub (specifically between closely-versioned releases), the upgrade process may hang or fail while deploying the SM meta-package.

This occurs because underlying PackageInstall (PKGI) resources enter a ReconcileFailed state due to transient registry unavailability early in the upgrade process. When the HubSM installer subsequently runs kapp deploy to roll out the new SM meta-package, kapp detects no configuration changes for packages that remain at the same version. Consequently, kapp does not re-apply those resources, leaving them in a failed state and causing the overall deployment to block indefinitely while waiting for a successful reconciliation.

Environment

Carvel 
Kapp 
Tanzu Hub
Tanzu Platform

Cause

This issue is caused by a race condition between the Kapp Controller restart, the Registry VM update, and the way kapp handles resource diffing:

 

• Transient Registry Errors: During the upgrade, Kapp Controller pods are restarted, and the Registry VM is updated with new OCI images. During this short transition window, some PackageInstall resources attempt to reconcile but fail due to transient errors (e.g., missing images or unavailable digests). These PKGIs immediately enter a ReconcileFailed state.

• Kapp Diffing Mechanism: Shortly after, the HubSM installer runs kapp deploy for the new version of the SM meta-package (which includes a newly versioned secret).

• Skipped Re-application: For packages whose version has not changed in this upgrade cycle, kapp detects no diff in the underlying Kubernetes manifests. Because there is no diff, kapp skips re-applying those specific PKGI resources.

• Blocked Deployment: Because the resources are not re-applied, no update event is sent to the Kubernetes API server. Kapp Controller is not forced to immediately re-reconcile them, leaving them in ReconcileFailed. The kapp deploy process blocks, waiting for these resources to succeed, relying entirely on kapp-controller's periodic background reconciliation to eventually retry and clear the error.

 

Resolution

To prevent the kapp deploy from hanging on these stale ReconcileFailed states, you must force Kapp Controller to immediately re-reconcile the PackageInstall resources before initiating the deployment.

This can be achieved by briefly pausing and unpausing all PackageInstall resources. Toggling the paused state updates the resource in the API server, which forces Kapp Controller to trigger a fresh reconciliation loop, clearing the transient failures before the SM meta-package deployment begins.

1. Pause the resources:

# Pause all PackageInstalls in the target namespace
kubectl patch pkgi -n tanzusm --all --type=merge -p '{"spec":{"paused":true}}'

2. Unpause the resources:

# Unpause all PackageInstalls to trigger immediate reconciliation
kubectl patch pkgi -n tanzusm --all --type=merge -p '{"spec":{"paused":false}}'