Unable to start vsan-health service on the vCenter server.

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

• Attempting to start vSAN service 'service-control --start vsan-health' fails with error : Service-control failed. Error: Failed to start srevices in profile ALL.RC=4, stderr=Failed to start vsan-health services. Error: A system error occurred. Check logs for details.
  
  
• On the vCenter server, /var/log/vmware/vsan-health/vsanvcmgmtd.log

xxxx-xx-xxTxx:xx:xx.xxxZ error vsanvcmgmtd[xxxxx] [vSAN@6876 sub=VsanMgmtServer] Failed to create key store, cert: '/etc/vmware-vpx/ssl/rui.crt', privateKey: '/etc/vmware-vpx/ssl/rui.key', Ex: N7Vmacore6Crypto15CryptoExceptionE(Crypto Exception: error:0200100D:system library:fopen:Permission denied: unable to load BIO) --> [context]zKq7AVECAQAAAHYmVQEadnNhbnZjbWdtdGQAAPbYN2xpYnZtYWNvcmUuc28AAI14LAALbC0AE+kyANJKKACZVSgAGFooABxcKAH8ugJ2c2FudmNtZ210ZAABSNYCAT1tAwIbTAlsaWJweXRob24zLjdtLnNvLjEuMAAChUwJAmdkBwIZ8BMCT0cJAnZmBwIZ8BMCXvETAovxEwJ/ihYCzUQYAtJGGAEo8QEDhysCbGliYy5zby42AAH18gE=[/context]
xxxx-xx-xxTxx:xx:xx.xxxZ warning vsanvcmgmtd[26478] [vSAN@6876 sub=VsanMgmtServer] Exit VsanMgmtServer::Init (1 ms)
xxxx-xx-xxTxx:xx:xx.000Z error vsanvcmgmtd[26478] [vSAN@6876 sub=Default] Failed to initialize vSAN management server: N7Vmacore6Crypto15CryptoExceptionE(Crypto Exception: error:0200100D:system library:fopen:Permission denied: unable to load BIO) --> [context]zKq7AVECAQAAAHYmVQEadnNhbnZjbWdtdGQAAPbYN2xpYnZtYWNvcmUuc28AAI14LAALbC0AE+kyANJKKACZVSgAGFooABxcKAH8ugJ2c2FudmNtZ210ZAABSNYCAT1tAwIbTAlsaWJweXRob24zLjdtLnNvLjEuMAAChUwJAmdkBwIZ8BMCT0cJAnZmBwIZ8BMCXvETAovxEwJ/ihYCzUQYAtJGGAEo8QEDhysCbGliYy5zby42AAH18gE=[/context]
xxxx-xx-xxTxx:xx:xx.xxxZ warning vsanvcmgmtd[26478] [vSAN@6876 sub=Default] Quick exit vsanvcmgmtd!

Environment

vCenter 7.x

vCenter 8.x

Cause

Permission related error when attempting to access the SSL certificate and key files located at /etc/vmware-vpx/ssl/rui.crt and /etc/vmware-vpx/ssl/rui.key

Resolution

1. Login to vCenter server via SSH.

      2. Compare the file permissions of rui.crt and rui.key on both affected and healthy vCenter appliances:

           ls -ltrh /etc/vmware-vpx/ssl/

Example:

-rw------- 1 vpxd cis  1.7K date xxxx rui.key
-rw-r----- 1 vpxd cis  1.7K date  xxxx rui.crt

      3. Change the permission of the folder using chmod or chown command accordingly.

Example:

If the permissions or ownership of rui.key and rui.crt differ from a healthy vCenter appliance, use the following commands to correct them:

# Set correct permissions
chmod 600 rui.key
chmod 640 rui.crt

# Set correct ownership
chown vpxd:cis rui.key
chown vpxd:cis rui.crt

      4. Following the permission correction, reload the service:

          vmon-cli -U vsan-health -R root