"Error : no healthy upstream or 503 Service Unavailable when accessing vCenter Server"

Products

VMware vCenter Server

Issue/Introduction

vCenter Server services fail to start or authenticating fails because one or more required security certificates are expired. This results in the vSphere Client UI becoming inaccessible.

Symptoms:

vSphere Client UI hangs or spins indefinitely during login.
503 Service Unavailable error
no healthy upstream error
Error appears on login page: [500] An error occurred while fetching identity providers.
VAMI (Port 5480) reports: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired.

Additional Symptoms Reported

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x...] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server
Signing certificate is not valid
Warnings in the vCenter interface show that certificates are expiring soon
Services such as vpxd, vpxd-svcs, etc, fail to start after vCenter reboot
vCert Option 1 shows expired certificates such as Machine certificate, Solution User certificates, CA certificate(s), SMS certificates, data-encipherment certificate, and/or STS certificates.
VDT output shows expired certificates.

Environment

VMware vCenter Server 6.x / 7.x / 8.x / 9.x

Cause

This issue occurs when the Security Token Service (STS), Machine SSL, or Solution User certificates exceed their validity period

Resolution

Prerequisites

Note: Before proceeding, take a backup or create a powered-off virtual machine snapshot of the vCenter Server.
For standalone vCenter: Take a snapshot of the vCenter VM (recommended from the host).
For Enhanced Linked Mode (ELM): Power off all linked vCenter Servers simultaneously and take a snapshot of each node (recommended from the hosts). See VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.
For vCenter HA (VCHA): Remove (destroy) the VCHA configuration before taking snapshots.

Fixed in vCenter Server 7.0 Update 3p and higher / 8.0 Update 2 and higher

Take a powered-off virtual machine snapshot of the vCenter Server.
Connect to the vCenter Server Appliance via SSH as root.
Check certificate expiration by running: for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
Run the vCert tool to replace expired certificates. Use Option 1 to identify expired certs and Option 6 to reset all certificates if multiple stores are impacted.
Restart services: service-control --stop --all && service-control --start --all
Confirm all services are running: service-control --status --all

For vCenter 6.x (Windows or Appliance)

Use the certificate-manager utility:

vCenter Server Appliance: /usr/lib/vmware-vmca/bin/certificate-manager
Windows vCenter Server: C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager
Select Option 8 (Reset all Certificates) to regenerate all certificates with VMCA-signed certificates.
For detailed steps, see Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA

Additional Information

Impact/Risks:

If issues occur during certificate replacement, the vCenter Server may become inaccessible. Ensure you have a valid snapshot before proceeding.
The VMDIR LDAP directory may also fail to update properly, so it may need to be repaired. See Using the 'lsdoctor' Tool