Tanzu Hub tile upgrades intermittently fails with kapp error "unable to retrieve the complete list of server APIs"
Article ID: 444033
Updated On:
Products
Issue/Introduction
Tanzu Hub tile upgrades fail intermittently when kapp attempts to deploy the package repository. You might see errors like following snippet:
6:09:55AM: Deploy started (10s ago)
6:10:05AM: Deploy failed
| kapp: Error: unable to retrieve the complete list of server APIs: data.packaging.carvel.dev/v1alpha1: stale GroupVersion discovery: data.packaging.carvel.dev/v1alpha1 (possibly related issue: https://github.com/carvel-dev/kapp/issues/12)
This issue is seen in Tanzu Hub 10.4.1 and below
Environment
Tanzu Hub
Tanzu Platform
Carvel kapp
Cause
This issue is caused by a timeout mismatch between the kapp CLI and the kube-apiserver during a TLS certificate race condition. During the upgrade, kapp-controller restarts and generates a new self-signed certificate. The kube-apiserver attempts to fetch the discovery schema before it receives the new certificate, resulting in an x509 error. This error places the kapp-controller API into an exponential backoff queue, which prevents the API server from serving the kapp-controller schema for approximately one minute.
Meanwhile, the kapp CLI attempts its own discovery fetch, encounters the stale API, and enters a hardcoded retry loop that times out after only 10 seconds. Since 10 seconds is insufficient to outlast the API server's one-minute backoff period, kapp
Resolution
This issue will be addressed in the future releases of Hub and also will be updated in the Release Notes.
To resolve this issue, retry the Tanzu Hub tile upgrade deployment. Because the failure is caused by an intermittent race condition during the initial certificate rotation, subsequent deployment attempts typically succeed once the kube-apiserver backoff period has cleared and the certificate has been successfully processed.
Feedback
