Error : "no healthy upstream" or "503 Service Unavailable" when accessing vCenter Server

VMware vCenter Server

vCenter Server services fail to start or authentication fails because one or more required security certificates are expired. This results in the vSphere Client UI becoming inaccessible.
Symptoms include:
- vSphere Client UI hangs or spins indefinitely during login.
- "503 Service Unavailable" error.
- "no healthy upstream" error.
- Error appears on login page: "[500] An error occurred while fetching identity providers".
- VAMI (Port 5480) reports: "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired".
- "503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x...] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)".
- "[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server".
- "Signing certificate is not valid".
- Warnings in the vCenter interface show that certificates are expiring soon.
- Services such as vpxd, vpxd-svcs, etc, fail to start after vCenter reboot.
- vCert Option 1 shows expired certificates such as Machine certificate, Solution User certificates, CA certificate(s), SMS certificates, data-encipherment certificate, and/or STS certificates.
- VDT output shows expired certificates.

VMware vCenter Server 6.x / 7.x / 8.x

VMware vCenter 9.x

This issue occurs when the Security Token Service (STS), Machine SSL, or Solution User certificates exceed their validity period.

Prerequisites:

Before proceeding, take a backup or create a powered-off snapshot of the vCenter Server Virtual Machine.
For standalone vCenter: Take a snapshot of the vCenter VM (recommended from the host).
For Enhanced Linked Mode (ELM): Power off all linked vCenter Servers simultaneously and take a snapshot of each node (recommended from the hosts). See VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.
For vCenter HA (VCHA): Remove (destroy) the VCHA configuration before taking snapshots.

For vCenter Server 7.x/8.x and vCenter 9.x:

Connect to the vCenter Server Appliance via SSH as root.
Check certificate expiration by running the following command:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
Run the vCert tool to replace expired certificates. Use Option 1 to identify expired certs and Option 6 to reset all certificates if multiple stores are impacted.
Note: The certificates can also be replaced by using Certificate Manager utility on vCenter. Refer: Using vSphere Certificate Manager to Replace SSL Certificates
Restart services: service-control --stop --all && service-control --start --all
Confirm all services are running: service-control --status --all

For vCenter 6.x (Windows or Appliance)

Use the certificate-manager utility:

vCenter Server Appliance: /usr/lib/vmware-vmca/bin/certificate-manager
Windows vCenter Server: C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager
Select Option 8 (Reset all Certificates) to regenerate all certificates with VMCA-signed certificates.
For detailed steps, see Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA

Impact/Risks:

If issues occur during certificate replacement, the vCenter Server may become inaccessible. Ensure you have a valid snapshot before proceeding.
The VMDIR LDAP directory may also fail to update properly, so it may need to be repaired. See Using the 'lsdoctor' Tool