vSAN-Health Service times out and fails to start on vCenter server

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter vsan-health service won't start automatically
• Attempting to start the vsan-health service ends with the service timing out
• The vpxd.log shows the following during service start up

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDThh:mm:ss warning vpxd[322534] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f4d9860dcc0, h:54, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 52892'>>), e: 167773205(sslv3 alert certificate expired (SSL routines)), duration: 2msec

• The vmware-vsan-health-service.log shows the following during service start

/var/log/vmware/vsan-health/vmware-vsan-health-service.log

YYYY-MM-DDThh:mm:ss WARNING vsan-mgmt[323169] [VsanVcExtension::__init__ opID=noOpId] Failed to log into VC, retrying in 10 seconds
YYYY-MM-DDThh:mm:ss WARNING vsan-mgmt[323817] [connectionpool::urlopen opID=noOpId] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ResponseNotReady('Request-sent'))': /vsan/plugins/vsan-ui-repa/plugin.zip

• The vmon.log shows the following

/var/log/vmware/vmon/vmon.log

YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Service start operation timed out.
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service exited. Exit code 1
YYYY-MM-DDThh:mm:ss Er(02) host-318478 Service batch op START failed. Failed services: 'vsan-health'

• The vsanvcmgmtd.log show the following repeatedly during service start

/var/log/vmware/vsan-health/vsanvcmgmtd.log

YYYY-MM-DDThh:mm:ss info vsanvcmgmtd[323169] [vSAN@6876 sub=PyCppVmomi] Loaded system certificate from VECS.
YYYY-MM-DDThh:mm:ss info vsanvcmgmtd[323169] [vSAN@6876 sub=vmomi.soapStub[1] opId=d0be8afb] SOAP request returned HTTP failure; <<io_obj p:0x00007fdb5c045a20, h:15, <TCP '127.0.0.1 : 56578'>, <TCP '127.0.0.1 : 1080'>>, /extension-login/sdk>, method: loginExtensionByCertificate; code: 526(Invalid SSL Certificate); fault: (null)
YYYY-MM-DDThh:mm:ss.272Z warning vsanvcmgmtd[323169] [vSAN@6876 sub=Py2CppStub opId=d0be8afb] |- EExit LOCAL::vim.SessionManager.loginExtensionByCertificate (28 ms)

• The vCenter Server is using External CA signed Custom SSL certificate

 

Environment

vCenter Server 8.x

Cause

This issue occurs due to an broken chain or expired root/intermediate CA certificate in the vCenter server's /etc/vmware-vpx/ssl/rui.crt file.

Resolution

To resolve this issue perform the following:

• Take a backup snapshot on the vCenter Server vm. For more details refer Snapshot Best practices for vCenter Server Virtual Machines
• Verify that the custom SSL certificate is valid and does not have any chain inconsistency by running the below command:

openssl verify /etc/vmware-vpx/ssl/rui.crt

A good certificate with a consistent chain would give output 'OK' as found below:

root@vcsa01 [ /etc/vmware-vpx/ssl ]# openssl verify rui.crt
rui.crt: OK

Additional parameters like "-show_chain" , "-verbose" can be used. For more details, please refer to OpenSSL-Verify

• Replace the vCenter machine SSL certificate properly by following Replace vCenter Machine SSL certificate Custom Certificate Authority Signed Certificate

Additional Information

If the above does not resolve this issue, Please review the Certificates to make sure the certificate does not contain any IP addresses in the SAN as per KB - vCenter vsan-health service fails to start due to BoringSSL IP SAN validation error : https://knowledge.broadcom.com/external/article?articleNumber=454810