vSAN Health service fails to start after performing vCenter upgrade

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

1. vSAN Health service fails to start in a VCHA configuration after performing a vCenter upgrade.
2. In /var/log/vmware/vsan-health/vsanvp.log on vCenter Server you can see errors similar to:

YYYY-MM--DDTHH:MM:SS.108Z INFO vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446] 
Register VSAN VP to SMS service.
YYYY-MM--DDTHH:MM:SS.256Z ERROR vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446] 
Failed to register VSAN VP to sms service
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/ServiceInstance.py", line 81, in RegisterVsanVP
raise info.error
VsanVmomiPatcher.sms.fault.ProviderRegistrationFault: (sms.fault.ProviderRegistrationFault) {
msg = 'Validation of VMOMI server version got from provider failedCould not find version class for: null',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}

3. In /var/log/vmware/vsan-health/vmware-vsan-health-service.log on vCenter Server you see errors similar to:

YYYY-MM--DDTHH:MM:SS.108Z CRITICAL vsan-mgmt[MainThread] [VsanMgmtServer::UncaughtExcpetionHandler opID=noOpId] Traceback (most recent call last):
   File "/usr/lib/vmware-vpx/vsan-health/VsanMgmtServer.py", line 436, in <module>
    _VsanMgmtServer.Start()
 pyVmomi.VmomiSupport.vmodl.RuntimeFault: (vmodl.RuntimeFault) {
   msg = 'N7Vmacore23FileIONotFoundExceptionE: Could not find file  : /etc/vmware-vsan-health/.cns_pgpass',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}

4. One may observe the following error when trying to add a VM Disk:

"PBM error occurred during PreReconfigureCallback: Fault cause: vmodl.fault.SystemError"

Environment

VMware vCenter Server 6.7.x

Cause

The issue observed on the SPBM side is occurring due to vSAN VP inability to register itself with the SMS, service since vmware-vsan-health service is failing to start due to missing /etc/vmware-vsan-health/.cns_pgpass file.

Resolution

This is a known issue affecting vCenter upgrade - e.g., when upgrading from any 6.7 to 6.7 U3 or when upgrading from 6.7 U3 to any 6.7 U3 patch release (e.g. 6.7 U3g).

Fix due in a future release, subscribe to this KB to be updated when the fixed in version is released.

Workaround:
To work around this issue, you must regenerate the missing file via the following procedure:

1. Login via SSH to the vCenter Appliance which failed to start vsan-health service after upgrade to 6.7 U3.
2. Run the following command to re-generate the missing file .cns_pgpass:

# PYTHONPATH=/usr/lib/vmware-vpx/pyJack/:/usr/lib/vmware/site-packages/  python /usr/lib/vmware-vpx/firstboot/CnsVcConfigure.py

3. You should see the following output in the terminal: 

YYYY-MM--DDTHH:MM:SS.108Z  Configuring DB for CNS service
YYYY-MM--DDTHH:MM:SS.108Z  Getting value for install-parameter: vpostgres.int.server_port
YYYY-MM--DDTHH:MM:SS.108Z  CNS DB configured successfully.

4. Start vsan-health service by running command: 

# service-control --start vmware-vsan-health

5. Verify the vsan-health service has started. 

# service-control --status vsan-health

   6. If step 4 doesn't work then reboot vCenter