Unable to create container volumes if vsan-health service is stopped

Products

• VMware vSphere Kubernetes Service
• VMware Cloud Native Runtimes

Issue/Introduction

If vsan-health service is in a stopped or disabled provisioning PVCs would fail with the following message. 

29s         Warning   ProvisioningFailed     persistentvolumeclaim/<pvc name>     failed to provision volume with StorageClass "<storage class>": rpc error: code = Internal desc = failed to create volume. Error: POST "/vsanHealth": 503 Service Unavailable

Environment

vSphere Supervisor
vCenter Server

Cause

The CSI Driver checks that the vSAN health Service is up and running before proceeding with volume creation.
If the service vSAN health Service is in a stopped or disabled state at volume creation, the CSI driver will fail to create the volume.

The service is used to communicate between the CSI driver and vCenter. 

Resolution

Confirm that the service is not running. 
    service-control --status vsan-health
    Stopped:
     vsan-health

Check the status of the service

[ ~ ]# vmon-cli -s vsan-health
Name: vsan-health
Starttype: DISABLED
RunState: STOPPED
RunAsUser: vsan-health
CurrentRunStateDuration(ms): 82120
HealthState: UNHEALTHY
FailStop: FALSE
MainProcessId: N/A

If in disabled state, then re-enable the service
    vmon-cli -U vsan-health -S AUTOMATIC

Start the service
     service-control --start vsan-health