Failed to start vpxd-svcs services. Error: A system error occurred. Check logs for details. Failed to start services in profile ALL.

Products

• VMware vCenter Server

Issue/Introduction

service-control --start --all 

Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start vpxd-svcs services. Error: A system error occurred. Check logs for details

From /var/log/vmware/vmon/vmon.log, we see below mentioned error

YYYY-MM-DDT21:01:24.607Z In(05) host-#### Executing service batch op START. IgnoreFail=0, service count=40
YYYY-MM-DDT21:01:24.607Z In(05) host-#### <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
YYYY-MM-DDT21:01:33.347Z Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####     endpoint_registration_runner(logging_file)
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
.
.
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####   File "/usr/lib/python3.7/ssl.py", line 870, in _create
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####     self.do_handshake()
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####   File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####     self._sslobj.do_handshake()
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-#### ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
YYYY-MM-DDT21:01:33.347Z Wa(03)+ host-####
YYYY-MM-DDT21:01:33.398Z Er(02) host-#### <vpxd-svcs> Service pre-start command failed with exit code 1.
YYYY-MM-DDT21:01:33.398Z Er(02) host-#### Service batch op START failed. Failed services: 'vpxd-svcs'

OR in /var/log/vmware/vpxd/vpxd.log

The remote certificate has these problems
The host name does not match the subject name(s) in certifiate.)
Failed to connect to authz service
Failed to start VMware VirtualCenter

Environment

VMware vCenter Server 8.0

VMware vCenter Server 7.0

 

Cause

This is caused due to one or multiple vCenter certificates being expired

Resolution

Note: 

If the affected vCenter is standalone, take a valid snapshot of the vCenter Server.

If the affected vCenter is in ELM, take a power-off snapshot of all the vCenters.

• Validate all vCenter certificates Verify and resolve expired vCenter Server certificates using command line interface

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

• Once we determine which certificate is expired, we can use the fixcerts tool in the Replace certificates on vCenter server using the Fixcerts script below to renew expired certificate(s) using the command 

python <Path of the file>/fixcerts_3_2.py replace --certType expired_only

OR 

• To replace Machine SSL certificates Using vSphere Certificate Manager to Replace SSL Certificates