Error: "Failed to load Certificate Signing Requests" in SDDC Manager for vCenter Server

Products

VMware vCenter Server VMware SDDC Manager / VCF Installer

Issue/Introduction

If the vCenter Server UI becomes inaccessible or unresponsive, API communication with VMware SDDC Manager fails. As a result, the vCenter certificate management workflow in SDDC Manager becomes non-functional, preventing users from downloading CSRs or uploading and installing certificates.

The following behaviors and errors may be observed within the SDDC Manager and vCenter Server environments:

Attempts to "Download CSR" for vCenter Server in the SDDC Manager UI fails with error "Failed to load Certificate Signing Requests. Unable to get CSR. Ensure that the Generate CSRs workflow is triggered for ####"
The vCenter Server inventory does not load, or gets stuck at a blank screen after login.
vCenter services may report a "Running" status despite actual communication failures.

vmon.log shows the vsan-health service continuously failing health checks and being force-killed:

/var/log/vmware/vmon/vmon.log

YYYY-MM-DDThh:mm Wa(03) host-##### <vsan-health> Health of service failed. Health data:
YYYY-MM-DDThh:mm In(05) host-##### <vsan-health> Recover from service api health check failure. Fail count 1
YYYY-MM-DDThh:mm Wa(03) host-##### <vsan-health> Sysprocess clean stop timed out. Force kill. Pid #####
YYYY-MM-DDThh:mm Wa(03) host-##### <vsan-health> Service exited. Exit code 1

The vpxd.log shows failures to acquire SSO tokens.Because the service cannot successfully authenticate, it is unable to populate the inventory.

/var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDThh:mm error vpxd[####] [Originator@#### sub=SsoWrapper] [AcquireToken] AcquireToken exception: N9SsoClient22CommunicationExceptionE(Server returned error: 503 Service Unavailable)

sps.log shows session validation failures and offline VASA providers.

/var/log/vmware/vcenter-server/logs/sps/sps.log:

YYYY-MM-DDThh:mm [pool-27-thread-2] ERROR opId=sps-Main-584721-977 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: Unexpected status code: 503 occured as provider: http://localhost:1080/vsan/vasa/version.xml is offline
YYYY-MM-DDThh:mm:ss [pool-38-thread-3] ERROR opId=####com.vmware.vim.storage.common.security.CommonActivationValidator - Failed to validate session: the token has expired
YYYY-MM-DDThh:mm:ss [Timer-0] ERROR opId=sps-Main-709445-963 com.vmware.vim.sms.provider.vasa.cert.VmcaClientHandler - Exception in invocation of getVmcaSignedCertificate

To verify the Machine SSL certificate expiration date, establish an SSH connection to the vCenter Server and run the following command: for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Environment

VMware vCenter Server

VMware SDDC Manager

Cause

Triggered by an expired VASA provider certificate, resource deadlocks in vsan-health and vmware-sps prevent core services from acquiring SSO tokens, resulting in a blank inventory for vCenter and disabled certificate workflows in SDDC Manager.

Resolution

Take an offline snapshot or backup of all vCenter nodes in the SSO domain before proceeding. For more information refer: Snapshot Best Practices for vCenter Server Virtual Machines

To resolve this issue, you must renew the VASA storage provider certificate. If the vSphere Client UI is blank, first restart the services to clear the deadlock and gain access to the UI.

Step 1: Clear the Service Deadlock (If the vSphere Client UI is inaccessible)

Establish an SSH connection to the vCenter Server Appliance using the root account.
Restart all vCenter services using the following command: service-control --stop --all && service-control --start --all
Log in to the vSphere Client after the services start and proceed to Step 2.
Note: The system operates in a degraded state until the certificate is replaced.

Step 2: Renew the Expired Storage Provider Certificate

Log in to the vSphere Client.
Select the vCenter Server in the inventory and navigate to vCenter Sever > Configure > Storage Providers.
Review the Certificate Expiry column to locate the expired provider.
Select the expired vSAN VASA storage provider and click the Synchronize Storage Providers icon.
If synchronization fails, manually remove the provider to force the generation of a new certificate:
Note: For external VASA providers, please consult your storage vendor first to ensure it is not actively being used, and for specific instructions on removing the expired VMCA-signed certificate from the array itself before proceeding.
1. Select the expired storage provider and click Remove.
2. Restart the sps service by running command: service-control --restart vmware-sps
3. Allow a few moments for the system to process the removal. The vCenter Server automatically re-registers the internal vSAN VASA provider and issues a valid certificate.
Verify the Certificate Expiry column now displays a valid future date.