VKS Guest Cluster Upgrade from 1.32 to 1.33 Fails: "AddonsReconciled condition is not True" due to cert-manager x509 errors.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to upgrade a vSphere with Tanzu (VKS) Guest cluster fails with the following error:

error: clusters.cluster.x-k8s.io "<CLUSTER_NAME>" could not be patched: admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: upgrade cannot be initiated as cluster's AddonsReconciled condition is not True. Message: Addon Pinniped is not ready: Addon Secretgen-Controller is not ready: Addon Vsphere-Cpi is not ready: Addon Vsphere-Pv-Csi is not ready: Addon Antrea is not ready: Addon Gateway-Api is not ready: Addon Guest-Cluster-Auth-Service is not ready: Addon Metrics-Server is not ready: . Reason: ReconcileFailed

Describing the affected packages displays the following status:

Reconcile failed: the server is currently unable to handle the request (get packages.data.packaging.carvel.dev)

In the Kubernetes API server log, TLS certificate failure errors are observed:

E0219 10:15:24.374048 1 controller.go:146] "Unhandled Error" err=<Error updating APIService "v1alpha1.data.packaging.carvel.dev" with err: failed to download v1alpha1.data.packaging.carvel.dev: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: tls: failed to verify certificate: x509: certificate signed by unknown authority, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]> logger="UnhandledError"

In the cert-manager-cainjector pod logs (kubectl logs -n cert-manager cert-manager-cainjector-<ID>), synchronization errors are continuously logged for the packages:

E0219 02:40:39.656212    1 indexers.go:61] "unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \"prometheus-###-tls\" not found" logger="cert-manager" kind="customresourcedefinition" type="customresourcedefinition" secret="monitoring/prometheus-###-tls" certificate="monitoring/prometheus-###-tls"
E0219 02:40:39.656271    1 indexers.go:61] "unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \"grafana-###-tls\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="monitoring/grafana-###-tls" certificate="monitoring/grafana-###-tls"
E0219 02:40:39.656315    1 indexers.go:61] "unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \"grafana-###-tls\" not found" logger="cert-manager" kind="customresourcedefinition" type="customresourcedefinition" secret="monitoring/grafana-###-tls"

Environment

vSphere with Tanzu (VKS)

VMware vSphere Kubernetes Service

Cause

The cert-manager-cainjector component has entered a degraded state when it is unable to fetch and inject the necessary CA certificates for the cluster webhooks and Custom Resource Definitions (CRDs).

This failure to inject valid certificates breaks TLS trust between the Kubernetes API server and the Carvel package management API (v1alpha1.data.packaging.carvel.dev).

Without TLS trust, the core cluster addons cannot communicate with the API to successfully reconcile, halting any cluster upgrade operations.

Resolution

1. Connect to the affected cluster.

2. Restart the cert-manager-cainjector deployment using the following command:

kubectl rollout restart deployment -n cert-manager cert-manager-cainjector

3. Monitor the newly created cert-manager-cainjector pod to ensure it initializes without the "unable to fetch certificate" errors.

4. Once the certificates are successfully injected, the addons will automatically retry and enter a successful Reconciled state.

5. Proceed with the cluster upgrade once the AddonsReconciled condition returns to True.