vCenter Server inaccessible due to expired Solution User certificates

Products

VMware vCenter Server

Issue/Introduction

Accessing the vSphere Client results in a failure to load the login page. The following symptoms may be encountered:

The following errors appear in the vSphere Client:
- [500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator. Back to login screen
- When attempting to login: "Username and password are required"
Running "service-control --status --all" in vCenter Shell confirms multiple services that are in a Stopped or StartPending state, including vmware-vpxd, vmware-vpxd-svcs, and vmware-sps.
When you try to start the service using the command "service-control --start vmware-vpxd" you get a message similar to one below:
```
"localized": "An error occurred while starting service 'vpxd'"
```
Output from running the vCert Tool confirms Solution User certificates have expired:
Command to check certificate expiration shows NOT AFTER date of today or older (example in image below does not show an expired certificate).

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

In the /var/log/vmware/vpxd/vpxd.log, the following authentication errors are recorded:

error vpxd[######] [Originator@#### sub=SsoWrapper] [AcquireToken] AcquireToken exception: N#SsoClient##InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
warning vpxd[######] [Originator@#### sub=Authz] [ConnectAndLogin] Failed to loginBySamlToken: N#SsoClient##InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
error vpxd[######] [Originator@#### sub=Authorize] Failed to initialize authorizeManager

Environment

VMware vCenter Server 8.x

Cause

This issue occurs because the vCenter Server Solution User certificates (machine, vsphere-webclient, vpxd, vpxd-extension, and hvc) have expired. These internal certificates need to be renewed periodically, and are required for vCenter services to authenticate with the Security Token Service (STS) to acquire SAML tokens.

Resolution

To resolve the issue, renew the vCenter Server Solution User certificates using the vCert script with VMCA as the certificate authority. Follow the detailed steps outlined in the KB article: vCert - Scripted vCenter expired certificate replacement.

Take an offline snapshot of vCenter, or all vCenters if using Enhanced linked mode. For more information refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice. If there are multiple vCenters in enhanced linked mode with expired solution user certificates, they will need to be renewed on each appliance.
Download the vCert tool and upload it to the vCenter in the /root or /tmp directory. See vCert - Scripted vCenter expired certificate replacement for more information.
Unzip the vCert tool:
```
# unzip -q vCert-########.zip
```
Navigate to the new vCert directory:
```
# cd vCert-########
```
Run the vCert tool:
```
./vCert.py
```
Select option 3 - Manage certificates
Select option 2 - Solution User certificates
Select option 1 - Replace with VMCA signed certificates
Restart services when prompted.

Additional Information

Determining expired SSL certificates in vCenter Server

Contact Broadcom support