vSphere Client reports "no healthy upstream" due to expired certificates

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Attempting to login to vCenter Server(VCSA) using vSphere Client may return error "no healthy upstream"
• vSAN configuration menus and health checks are not visible in the vCenter UI if it is loading
• Hosts are reporting in maintenance mode, but unable to exit
• Authentication is failing on vCenter Server Appliance Management Interface (VAMI) with the error : 
  
  Exception in invoking authentication handler [SSL;CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
• Validating the service status using SSH to VCSA

 

# service-control --status --all

Running:
 applmgmt lookupsvc lwsmd observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmonapi vmware-certificateauthority vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-infraprofile vmware-postgres-archiver vmware-rhttppro
xy vmware-sca vmware-stsd vmware-trustmanagement vmware-vmon vmware-vpostgres vtsdb

Stopped:
 observability vmcam vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-topologysvc vmware-vapi-endpoint vmware-vcha vmware-vdtc vmware-vpxd-svcs vsphere-ui vstats vmware-analytics vmware-sps vmware-updatemgr vmware-vsan-health vmware-vsm wcp vmware-vpxd

 

• /var/log/vmware/vmon/vmon.log  on VCSA

 

YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr:     endpoint_registration_runner()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-<PID>   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr:     UpdateTaggingServiceGrpcEndpoint(logger).run()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-<PID>   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run

YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
YYYY-MM-DDTHH:MM:SS Er(02) host-<PID> <vpxd-svcs> Service pre-start command failed with exit code 1.

 

Cause

vCenter services fail to start after certificates expire because VCSA is reliant on authenticated communication between its internal services/components. When a critical certificate, especially the Machine SSL Certificate or the Secure Token Service (STS) Signing Certificate expires, the services cannot perform required trust checks during startup, leading to failure.

Resolution

Note: Take a backup or create a virtual machine snapshot before proceeding. In case of ELM, refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot best practice

1. Refer to vCert - Scripted vCenter expired certificate replacement to download the vCert utility and upload it to the VCSA.

2. Connect to the VCSA on SSH as the root user and navigate to the directory where vCert utility was uploaded.

3. Run the vCert utility : python vCert.py

4. Select the appropriate replacement option based on the type of the current Machine SSL certificate:

• • To validate the status of all the certificates
    
    
    • Option 1: Check current certificate status
      
      
  • To replace Machine SSL Certificate
    
    
    • Option 3: Manage certificates → Option 1: Machine SSL certificate → Option 1. Replace Machine SSL certificate with a VMCA-signed certificate
      
      
  • To replace Solution User Certificate
    
    
    • Option 3: Manage certificates → Option 2. Solution User certificates → Option 1. Replace Solution User certificate with a VMCA-signed certificate

Note: In case of multiple certificates are reported as expired, proceed with Option 6. Reset all certificates with VMCA-signed certificates