Unable to start vsan-health service on the vCenter server.

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

• Attempting to start vSAN service 'service-control --start vsan-health' fails with error : Service-control failed. Error: Failed to start srevices in profile ALL.RC=4, stderr=Failed to start vsan-health services. Error: A system error occurred. Check logs for details.
  
  
• Scheduled vCenter Server backups may fail
• On the vCenter server, /var/log/vmware/vsan-health/vsanvcmgmtd.log

xxxx-xx-xxTxx:xx:xx.xxxZ error vsanvcmgmtd[xxxxx] [vSAN@6876 sub=VsanMgmtServer] Failed to create key store, cert: '/etc/vmware-vpx/ssl/rui.crt', privateKey: '/etc/vmware-vpx/ssl/rui.key', Ex: N7Vmacore6Crypto15CryptoExceptionE(Crypto Exception: error:0200100D:system library:fopen:Permission denied: unable to load BIO) --> [context]zKq7AVECAQAAAHYmVQEadnNhbnZjbWdtdGQAAPbYN2xpYnZtYWNvcmUuc28AAI14LAALbC0AE+kyANJKKACZVSgAGFooABxcKAH8ugJ2c2FudmNtZ210ZAABSNYCAT1tAwIbTAlsaWJweXRob24zLjdtLnNvLjEuMAAChUwJAmdkBwIZ8BMCT0cJAnZmBwIZ8BMCXvETAovxEwJ/ihYCzUQYAtJGGAEo8QEDhysCbGliYy5zby42AAH18gE=[/context]
xxxx-xx-xxTxx:xx:xx.xxxZ warning vsanvcmgmtd[26478] [vSAN@6876 sub=VsanMgmtServer] Exit VsanMgmtServer::Init (1 ms)
xxxx-xx-xxTxx:xx:xx.000Z error vsanvcmgmtd[26478] [vSAN@6876 sub=Default] Failed to initialize vSAN management server: N7Vmacore6Crypto15CryptoExceptionE(Crypto Exception: error:0200100D:system library:fopen:Permission denied: unable to load BIO) --> [context]zKq7AVECAQAAAHYmVQEadnNhbnZjbWdtdGQAAPbYN2xpYnZtYWNvcmUuc28AAI14LAALbC0AE+kyANJKKACZVSgAGFooABxcKAH8ugJ2c2FudmNtZ210ZAABSNYCAT1tAwIbTAlsaWJweXRob24zLjdtLnNvLjEuMAAChUwJAmdkBwIZ8BMCT0cJAnZmBwIZ8BMCXvETAovxEwJ/ihYCzUQYAtJGGAEo8QEDhysCbGliYy5zby42AAH18gE=[/context]
xxxx-xx-xxTxx:xx:xx.xxxZ warning vsanvcmgmtd[26478] [vSAN@6876 sub=Default] Quick exit vsanvcmgmtd!

• You may find vSAN-Health Service fails to start with stack below
  
  

  > service-control --start vmware-vsan-health

  Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsan-health...

  Error executing start on service vsan-health. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
        "args"; [
        "vsan-health"
        ],
            "localized": "An error occurred while starting service 'vsan-health'"
        }
    ],
    "componentKey": null
    "problemId": null,
    "resolution": null
  }
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vsan-health"
            ],
            "localized": "An error occurred while starting service 'vsan-health'"
        }
    ],
    "componentKey": null
    "problemId": null,
    "resolution": null,
}

Environment

vCenter 7.x

vCenter 8.x

Cause

Permission related error when attempting to access the SSL certificate and key files located at /etc/vmware-vpx/ssl/rui.crt and /etc/vmware-vpx/ssl/rui.key

Resolution

1. Login to vCenter server via SSH.

      2. Compare the file permissions of rui.crt and rui.key on both affected and healthy vCenter appliances:

           ls -ltrh /etc/vmware-vpx/ssl/

Example:

-rw------- 1 vpxd cis  1.7K date xxxx rui.key
-rw-r----- 1 vpxd cis  1.7K date  xxxx rui.crt

      3. Change the permission of the folder using chmod or chown command accordingly.

• Example:

If the permissions or ownership of rui.key and rui.crt differ from a healthy vCenter appliance, use the following commands to correct them:

# Set correct permissions
chmod 600 rui.key
chmod 640 rui.crt

# Set correct ownership
chown vpxd:cis rui.key
chown vpxd:cis rui.crt

• For vCenter 7.0 Builds 24322018 or newer (not vCenter 8.x/9.x):

# Set correct permissions
chmod 640 rui.key
chmod 640 rui.crt

# Set correct ownership
chown root:cis rui.key
chown root:cis rui.crt

      4. Following the permission correction, reload the service:

          vmon-cli -U vsan-health -R root