vSAN Health service fails to start after performing vCenter upgrade
search cancel

vSAN Health service fails to start after performing vCenter upgrade

book

Article ID: 326836

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  1. vSAN Health service fails to start in a VCHA configuration after performing a vCenter upgrade.
  2. In /var/log/vmware/vsan-health/vsanvp.log on vCenter Server you can see errors similar to:
YYYY-MM--DDTHH:MM:SS.108Z INFO vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446] 
Register VSAN VP to SMS service.
YYYY-MM--DDTHH:MM:SS.256Z ERROR vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446] 
Failed to register VSAN VP to sms service
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/ServiceInstance.py", line 81, in RegisterVsanVP
raise info.error
VsanVmomiPatcher.sms.fault.ProviderRegistrationFault: (sms.fault.ProviderRegistrationFault) {
msg = 'Validation of VMOMI server version got from provider failedCould not find version class for: null',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
  1. In /var/log/vmware/vsan-health/vmware-vsan-health-service.log on vCenter Server you see errors similar to:
YYYY-MM--DDTHH:MM:SS.108Z CRITICAL vsan-mgmt[MainThread] [VsanMgmtServer::UncaughtExcpetionHandler opID=noOpId] Traceback (most recent call last):
   File "/usr/lib/vmware-vpx/vsan-health/VsanMgmtServer.py", line 436, in <module>
    _VsanMgmtServer.Start()
 pyVmomi.VmomiSupport.vmodl.RuntimeFault: (vmodl.RuntimeFault) {
   msg = 'N7Vmacore23FileIONotFoundExceptionE: Could not find file  : /etc/vmware-vsan-health/.cns_pgpass',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}
  1. One may observe the following error when trying to add a VM Disk:
"PBM error occurred during PreReconfigureCallback: Fault cause: vmodl.fault.SystemError"



Environment

VMware vCenter Server 6.7.x

Cause

The issue observed on the SPBM side is occurring due to vSAN VP inability to register itself with the SMS, service since vmware-vsan-health service is failing to start due to missing /etc/vmware-vsan-health/.cns_pgpass file.

Resolution

This is a known issue affecting vCenter upgrade - e.g., when upgrading from any 6.7 to 6.7 U3 or when upgrading from 6.7 U3 to any 6.7 U3 patch release (e.g. 6.7 U3g).

Fix due in a future release, subscribe to this KB to be updated when the fixed in version is released.

Workaround:
To work around this issue, you must regenerate the missing file via the following procedure:

  1. Login via SSH to the vCenter Appliance which failed to start vsan-health service after upgrade to 6.7 U3.
  2. Run the following command to re-generate the missing file .cns_pgpass:
# PYTHONPATH=/usr/lib/vmware-vpx/pyJack/:/usr/lib/vmware/site-packages/  python /usr/lib/vmware-vpx/firstboot/CnsVcConfigure.py
  1. You should see the following output in the terminal: 
YYYY-MM--DDTHH:MM:SS.108Z  Configuring DB for CNS service
YYYY-MM--DDTHH:MM:SS.108Z  Getting value for install-parameter: vpostgres.int.server_port
YYYY-MM--DDTHH:MM:SS.108Z  CNS DB configured successfully.
  1. Start vsan-health service by running command: 
# service-control --start vmware-vsan-health
  1. Verify the vsan-health service has started. 
# service-control --status vsan-health
   6. If step 4 doesn't work then reboot vCenter