Supervisor Upgrade Fails at ImageRegistryUpgrade With Error "Certificate Unknown Authority"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• During an upgrade of a Supervisor Cluster, the process hangs or fails at the ImageRegistryUpgrade component.
• The vSphere UI or Workload Control Plane (WCP) logs report the following error:

Component Configuration error: Component ImageRegistryUpgrade failed: Failed to run command: [...] Internal error occurred: failed calling webhook "default.validating.namespace.supervisor.vmware.com": failed to call webhook: Post "https://vmware-system-nsop-webhook-service.vmware-system-nsop.svc:443/supervisor-namespace-validate-v1-namespace?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority Component upgrade failed.

Environment

• VMware vSphere Kubernetes Service

Cause

• The Kubernetes API server is unable to validate the TLS certificate presented by the Namespace Operator Validating Webhook (vmware-system-nsop-webhook-service) when applying namespace patches during the upgrade. This is caused by a synchronization issue within cert-manager, resulting in either an invalid TLS certificate being generated for the webhook service or the caBundle temporarily falling out of sync.

Resolution

Before proceeding with command-line interventions, attempt to click Apply again in the vSphere UI (Workload Management -> Updates). Certificate synchronization sometimes resolves organically upon a retry. If the upgrade fails again, proceed below.

Recommended: Refresh cert-manager certificates 

1. Retrieve the root password for the Supervisor Control Plane nodes from vCenter.

2. SSH into one of the Supervisor Control Plane VMs.

3. Restart the cert-manager deployments:
   
   kubectl rollout restart deployment -n vmware-system-cert-manager cert-manager-cainjector
kubectl rollout restart deployment -n vmware-system-cert-manager cert-manager
kubectl rollout restart deployment -n vmware-system-cert-manager cert-manager-webhook

4. Verify that the new pods have successfully started: kubectl get pods -n vmware-system-cert-manager

5. SSH into the vCenter Server Appliance (VCSA) as root.

6. Restart the WCP service to clear any cached states: service-control --restart wcp

7. Return to the vSphere Client UI and re-initiate the Supervisor upgrade.

Workaround: Temporarily bypass the webhook  If refreshing the certificates does not resolve the issue, instruct the API server to ignore the certificate failure for this specific webhook.

1. SSH into a Supervisor Control Plane node.

2. Edit the validating webhook configuration for NSOP: kubectl edit validatingwebhookconfiguration vmware-system-nsop-validating-webhook-configuration

3. Locate the failurePolicy field under the webhooks definition (currently set to Fail).

4. Modify the value to Ignore.

5. Save the file and exit the editor.

6. Return to the vSphere Client UI and restart the upgrade.

7. Once the upgrade completes, restore the webhook definition to Fail.