vSphere with Tanzu Supervisor upgrade fails with error "webhook.cert-manager.io" denied the request: spec.selfSigned: Forbidden: may not specify more than one issuer type Component upgrade failed.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During a vSphere with Tanzu (WCP) Supervisor cluster upgrade, the upgrade process halts and the cluster remains in an upgrading state.

Reviewing the WCP controller logs (/var/log/vmware/wcp/wcpsvc.log) reveals the following Component upgrade failed error:

Error from server (Forbidden): error when applying patch: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"cert-manager.io/v1\",\"kind\":\"Issuer\",\"metadata\":{\"annotations\":{\"kubernetes.io/change-cause\":\"kubectl apply --filename=/usr/lib/vmware-wcp/objects/pinniped/supervisor --record=true\"},\"name\":\"pinniped-supervisor-ca-issuer\",\"namespace\":\"vmware-system-pinniped\"},\"spec\":{\"ca\":{\"secretName\":\"pinniped-supervisor-root-ca-secret\"}}}\n","kubernetes.io/change-cause":"kubectl apply --filename=/usr/lib/vmware-wcp/objects/pinniped/supervisor --record=true"}},"spec":{"ca":{"secretName":"pinniped-supervisor-root-ca-secret"}}} to: Resource: "cert-manager.io/v1, Resource=issuers", GroupVersionKind: "cert-manager.io/v1, Kind=Issuer" Name: "pinniped-supervisor-ca-issuer", Namespace: "vmware-system-pinniped" for: "/usr/lib/vmware-wcp/objects/pinniped/supervisor/supervisor.yaml": error when patching "/usr/lib/vmware-wcp/objects/pinniped/supervisor/supervisor.yaml": admission webhook "webhook.cert-manager.io" denied the request: spec.selfSigned: Forbidden: may not specify more than one issuer type Component upgrade failed.admission webhook "webhook.cert-manager.io" denied the request: spec.selfSigned: Forbidden: may not specify more than one issuer type Component upgrade failed.

Additionally, missing annotation warnings may precede the failure: Warning: resource issuers/pinniped-supervisor-ca-issuer is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply.

Environment

VMware Kubernetes Service 8.x, 9.x

Cause

This issue is caused by a Kubernetes Strategic Merge Patch conflict during the WCP declarative upgrade workflow.

The existing Pinniped issuer resources (pinniped-supervisor-selfsigned-issuer or pinniped-supervisor-ca-issuer) in the vmware-system-pinniped namespace contain legacy field ownership (managedFields) tied to kubectl-client-side-apply. This locks the selfSigned or ca configuration parameters. Because the declarative 3-way merge history is broken (missing last-applied-configuration annotations), the Kubernetes API attempts to merge the incoming upgrade manifest without automatically dropping the locked legacy fields.

The cert-manager webhook evaluates this malformed merged object in memory, detects multiple issuer types defined simultaneously, and rejects the API request, halting the upgrade.

Resolution

To resolve the merge conflict, you must manually delete the corrupted issuer objects. The declarative WCP controller will automatically recreate them cleanly from the updated manifests.

1. SSH into the vCenter Server Appliance (VCSA) as root.

2. Back up the existing issuers before proceeding:
   
   kubectl get issuer pinniped-supervisor-selfsigned-issuer pinniped-supervisor-ca-issuer -n vmware-system-pinniped -o yaml > /tmp/pinniped-issuers-backup.yaml

3. Delete the conflicting Pinniped issuers:
   
   kubectl delete issuer pinniped-supervisor-selfsigned-issuer pinniped-supervisor-ca-issuer -n vmware-system-pinniped --force --grace-period=0

4. Restart the WCP service to force an immediate reconciliation loop:
   
   service-control --restart wcp

5. Verify the WCP controller has successfully recreated the issuers:
   
   kubectl get issuers -n vmware-system-pinniped

6. Monitor the vSphere Client UI or wcpsvc.log to confirm the Supervisor upgrade successfully resumes and progresses.

 

If the WCP controller fails to recreate the deleted objects and the cluster degrades, you can revert to the pre-deletion state using the backup file created in Step 2.

1. Reapply the backed-up YAML configuration: kubectl apply -f /tmp/pinniped-issuers-backup.yaml (Note: If the restore fails with a resourceVersion conflict, open /tmp/pinniped-issuers-backup.yaml in a text editor, remove the resourceVersion and uid lines from the metadata of each object, and run the apply command again).

2. Take note of the error in the upgrade stage again if the same error is thrown and contact GS for next steps in that particular situation.

Additional Information

This behavior is specific to environments where historical upgrade paths or manual imperative changes (kubectl edit, kubectl patch) stripped the necessary declarative annotations from the cert-manager objects.