vCenter Server UI fails to Load with "No healthy upstream" or "[500] An error occurred while fetching identity providers" due to expired solution user certificates

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to access the vCenter Server UI through a browser using VMCA-signed certificates, one of the following errors is encountered:

• [500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator.
• No healthy upstream

• After logging in on the VAMI page(i.e <vc_fqdn>:5480) as root or administrator the health status is observed to be unknown as seen below:

 

• Upon checking the certificate expiry status using the command below, it is found that one or more Solution User certificates on the vCenter Server has expired.
  • for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Example output:

 

Environment

 VMware vCenter Server 7.x

 VMware vCenter Server 8.x

Cause

• The issue occurs due to expired Solution User certificates in the vCenter Server.
• These certificates are essential for authentication and secure communication between vCenter services.
• Once expired, the internal services such as STS (Security Token Service) and SSO (Single Sign-On) cannot properly authenticate, leading to a 500 or no healthy upstream error on the UI.

Resolution

Note: Take an appropriate snapshot of the vCenter server virtual machine, referring to Snapshot Best practices for vCenter Server Virtual Machines

To resolve the issue, renew the vCenter Server Solution User certificates with VMCA as the certificate authority by following either of the 2 options:

Option 1: To use Certificate Manager utility

1. Launch the vSphere 6/7/8.x Certificate Manager by executing the following command in SSH of the vCenter Server,
   
   For vCenter Server 6/7/8.x Appliance:
   
   /usr/lib/vmware-vmca/bin/certificate-manager
    
2. Select Option 6 (Replace Solution user certificates with VMCA Certificates)
    
3. Type Yes (Y) to the confirmation request to proceed.
4. Provide the [email protected] password when prompted. 

 For further details refer How to replace the vCenter Server Solution User certificates with VMCA issued certificate

Option 2: To use vCert script:

1. Download and upload the vCert script to the vCenter Server as per the KB: vCert script
2. Run the script as ./vCert.py and use option 3 to Manage certificates
3. Use option 2 to replace Solution user certificates with VMCA certificate
4. Restart the vCenter Server services

Additional Information

Verify and resolve expired vCenter Server certificates using the command line interface