vSAN-Health Service times out and fails to start on vCenter server

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter vsan-health service won't start automatically
• Attempting to start the vsan-health service ends with the service timing out
• The vpxd.log shows the following during service start up

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDThh:mm:ss warning vpxd[322534] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f4d9860dcc0, h:54, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 52892'>>), e: 167773205(sslv3 alert certificate expired (SSL routines)), duration: 2msec

• The vmware-vsan-health-service.log shows the following during service start

/var/log/vmware/vsan-health/vmware-vsan-health-service.log

YYYY-MM-DDThh:mm:ss WARNING vsan-mgmt[323169] [VsanVcExtension::__init__ opID=noOpId] Failed to log into VC, retrying in 10 seconds
YYYY-MM-DDThh:mm:ss WARNING vsan-mgmt[323817] [connectionpool::urlopen opID=noOpId] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ResponseNotReady('Request-sent'))': /vsan/plugins/vsan-ui-repa/plugin.zip

• The vmon.log shows the following

/var/log/vmware/vmon/vmon.log

YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service api-health command's stderr: ERROR:root:Got URL error HTTP Error 503: Service Unavailable
YYYY-MM-DDThh:mm:ss Wa(03)+ host-318478
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-318478 <vsan-health> Service start operation timed out.
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
YYYY-MM-DDThh:mm:ss Wa(03) host-318478 <vsan-health> Service exited. Exit code 1
YYYY-MM-DDThh:mm:ss Er(02) host-318478 Service batch op START failed. Failed services: 'vsan-health'

• The vsanvcmgmtd.log show the following repeatedly during service start

/var/log/vmware/vsan-health/vsanvcmgmtd.log

YYYY-MM-DDThh:mm:ss info vsanvcmgmtd[323169] [vSAN@6876 sub=PyCppVmomi] Loaded system certificate from VECS.
YYYY-MM-DDThh:mm:ss info vsanvcmgmtd[323169] [vSAN@6876 sub=vmomi.soapStub[1] opId=d0be8afb] SOAP request returned HTTP failure; <<io_obj p:0x00007fdb5c045a20, h:15, <TCP '127.0.0.1 : 56578'>, <TCP '127.0.0.1 : 1080'>>, /extension-login/sdk>, method: loginExtensionByCertificate; code: 526(Invalid SSL Certificate); fault: (null)
YYYY-MM-DDThh:mm:ss.272Z warning vsanvcmgmtd[323169] [vSAN@6876 sub=Py2CppStub opId=d0be8afb] |- EExit LOCAL::vim.SessionManager.loginExtensionByCertificate (28 ms)

• The vCenter Server is using External CA signed Custom SSL certificate

 

Environment

vCenter Server 8.x

Cause

This issue occurs due to an broken chain or expired root/intermediate CA certificate in the vCenter server's /etc/vmware-vpx/ssl/rui.crt file.

Resolution

To resolve this issue perform the following:

• Take a backup snapshot on the vCenter Server vm. For more details refer Snapshot Best practices for vCenter Server Virtual Machines
• Verify that the custom SSL certificate is valid and does not have any chain inconsistency by running the below command:

openssl verify /etc/vmware-vpx/ssl/rui.crt

A good certificate with a consistent chain would give output 'OK' as found below:

root@vcsa01 [ /etc/vmware-vpx/ssl ]# openssl verify rui.crt
rui.crt: OK

Additional parameters like "-show_chain" , "-verbose" can be used. For more details, please refer to OpenSSL-Verify

• Replace the vCenter machine SSL certificate properly by following Replace vCenter Machine SSL certificate Custom Certificate Authority Signed Certificate