WCP Service repeatedly crashes after rotating Machine SSL certificates and Supervisor Missing in vCenter

Products

VMware vCenter Server

Issue/Introduction

The Supervisor Management page in vCenter throws an error and fails to display supervisor details.
VCF Operations generates alerts indicating objects are not receiving data from adapter instances.
The WCP service (wcpsvc) is in a stopped or crashing state, and attempts to restart it fail.

The occurrence of a cascading failure followed by a stack overflow can be verified in the system logs:

Content Library authentication failure (cls.log):

YYYY-MM-DDTHH:MM:SSZ INFO CLS ## [vc@#### threadName="tomcat-http-2" logger="SessionAuthnHandlerImpl"]  Unauthenticated error while attempting to retrieve >valid vAPI session. This is non-fatal and the operation will be retried.YYYY-MM-DDTHH:MM:SSZ INFO CLS ## [vc@#### threadName="tomcat-http-2" logger="AuthenticationFilter"]  Authentication failedcom.vmware.vapi.std.errors.Unauthenticated: Unauthenticated

vAPI gateway connection rejection (endpoint.log):

com.vmware.vapi.client.exception.HttpProtocolException: HTTP response with status code 503 (enable debug logging for details): no healthy upstream

WCP fallback failure on missing local images (wcpsvc.log): Example version: v1.32.9+vmware.2-fips-vsc9.1.0.0-25370922

YYYY-MM-DDTHH:MM:SSZ ERROR wcp 2841914 [vc@####] [kubelifecycle/kube_instance.go:####] [opID=#######-####-####-####-#######dff-host-##->vLCM:Enable:domain-c##] Unable to find image info of desired version: v1.32.9+vmware.2-fips-vsc9.1.0.0-25370922 error: target Supervisor version not found for the >desired version: v1.32.9+vmware.2-fips-vsc9.1.0.0-25370922 in source type: EMBEDDED

WCP recursive retry crashing the service (stdstream.log.stderr):

runtime: goroutine stack exceeds 1000000000-byte limitruntime: sp=0xc03bc60370 stack=[0xc03bc60000, 0xc05bc60000]fatal error: stack overflow

Environment

VMware vCenter Server 9.1.0

VMware Cloud Foundation 9.1.0

vSphere Supervisor

Air-gapped / Offline environments

Cause

The issue is triggered by a certificate update made on the vCenter (VC) followed by an incorrect or missing service restart. This oversight introduces replication and synchronization latencies within the Single Sign-On (SSO) directory causing the Content Library Service to fail its authentication handshake with the internal vAPI gateway.

When the vAPI gateway rejects the connection, the WCP service cannot query the Content Library for required Supervisor component images and falls back to search local storage. In an air-gapped environment without local files, WCP triggers an internal error retry handler. A software defect in this retry mechanism causes a recursive loop without backoff, exhausting the call-stack memory and resulting in a stack overflow that crashes the WCP service.

Resolution

A code fix resolving the recursive WCP service crash behavior is planned for a future VCF 9.1 patch release.

To mitigate the issue in the current version, a strict sequential restart of the management services is required, followed by a validation of the database mapping:

Clear the authentication cache by performing a chained restart of the vCenter authentication and Content Library service stack:

service-control --stop wcp vmware-content-library vapi-endpoint 

service-control --start vapi-endpoint 

service-control --start vmware-content-library 

service-control --start wcp

If the WCP service continues to crash or fail to initialize, the database cluster configurations may need to be strictly aligned to the active Content Library Container UUID. Please contact Broadcom Support for assistance in verifying and safely updating the vCenter Database (VCDB).
Once the WCP service is stable, reboot any ESXi host nodes stuck in Maintenance Mode with inactive spherelet services to re-establish TLS handshakes.