vmware-vpxd-svcs and dependent servics are stopped due to [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)

Products

• VMware vCenter Server 8.0

Issue/Introduction

Attempting to login to vCenter Server using vSphere Client may return error "no healthy upstream" or VMware vSphere web client has stopped responding.

vCenter won't come up after a reboot. VAMI at port 5480 is unavailable.

 

After SSH into the vCenter VM, the following error may be displayed:

Error failed to connect to service. use service-control command to manage applmgmt service

 

Validating the service status using ssh to vCenter server VM shows many system critical services in stopped state, including vsphere-ui.

# service-control --status --all

Running:
 applmgmt lookupsvc lwsmd observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmonapi vmware-certificateauthority vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-infraprofile vmware-postgres-archiver vmware-rhttppro
xy vmware-sca vmware-stsd vmware-trustmanagement vmware-vmon vmware-vpostgres vtsdb

Stopped:
 observability vmcam vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-topologysvc vmware-vapi-endpoint vmware-vcha vmware-vdtc vmware-vpxd-svcs vsphere-ui vstats vmware-analytics vmware-sps vmware-updatemgr vmware-vsan-health vmware-vsm wcp vmware-vpxd

 

Viewing the /var/log/vmware/vmon/vmon.log  on vCenter Server VM shows errors similar to the following indicating certificate expiry:

YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr:     endpoint_registration_runner()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-<PID>   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr:     UpdateTaggingServiceGrpcEndpoint(logger).run()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-<PID>   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run

YYYY-MM-DDTHH:MM:SS Wa(03) host-<PID> <vpxd-svcs> Service pre-start command's stderr: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
YYYY-MM-DDTHH:MM:SS Er(02) host-<PID> <vpxd-svcs> Service pre-start command failed with exit code 1.

 

Environment

VMware vCenter Server

Cause

vCenter services fail to start after certificates expire because vCenter Server is reliant on authenticated communication between its internal services/components. When a critical certificate, especially the Machine SSL Certificate or the Secure Token Service (STS) Signing Certificate expires, the services cannot perform required trust checks during startup, leading to failure.

Resolution

Note: Take a backup or create a virtual machine snapshot before proceeding. In case of ELM, refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot best practice

1. Refer to vCert - Scripted vCenter expired certificate replacement to download the vCert utility and upload it to the vCenter server.

2. Run the vCert tool on the vCenter appliance and Select the appropriate replacement option based on the type of the current Machine SSL certificate:

• • To validate the status of all the certificates
    
    
    • Option 1: Check current certificate status
      
      
  • To replace Machine SSL Certificate
    
    
    • Option 3: Manage certificates → Option 1: Machine SSL certificate → Option 1. Replace Machine SSL certificate with a VMCA-signed certificate
      
      
  • To replace Solution User Certificate
    
    
    • Option 3: Manage certificates → Option 2. Solution User certificates → Option 1. Replace Solution User certificate with a VMCA-signed certificate

Note: In case of multiple certificates are reported as expired, proceed with Option 6. Reset all certificates with VMCA-signed certificates