vmware-vsan-health service fails to start after update to vCenter 6.7U3 with Error : Validation of VMOMI server version got from provider failed

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After udpating VCSA to 6.7U3, vsan-health service fails to start.
• The VAMI based backup fails mentioning that 'vsan-health' service is not running

sps.log:

2019-11-18T09:20:43.843-06:00 [pool-18-thread-5] INFO  opId=W1 com.vmware.vim.sms.StorageManagerImpl - Timer stopped: queryProvider, Time taken: 0 ms.
2019-11-18T09:20:43.937-06:00 [pool-12-thread-2] INFO  opId= com.vmware.vim.sms.StorageManagerImpl - Starting Timer: registerProviderInt. Provider spec: (sms.provider.VmodlVasaProviderSpec) {
   dynamicType = null,
   dynamicProperty = null,
   name = VMware vSAN,
   description = vSAN VASA 3.0 provider,
   url = https://VCENTER.dommain.local:443/vsanHealth/vsanvp/version.xml,
   authenticationType = UseSessionId
}
2019-11-18T09:20:43.951-06:00 [pool-12-thread-2] WARN  opId= com.vmware.vim.sms.provider.vasa.VersionHandler - [isLegacyProvider] Failed to retrive version information from provider - https://VCENTER.cm
css.net:443/vsanHealth/vsanvp/version.xml: Server returned HTTP response code: 503 for URL: https://VCENTER.domain.local:443/vsanHealth/vsanvp/version.xml
2019-11-18T09:20:43.951-06:00 [pool-12-thread-2] WARN  opId= com.vmware.vim.sms.provider.vasa.VersionHandler - [isLegacyProvider] Failed to retrive version information from provider. This could be a 1.0
 provider.
2019-11-18T09:20:43.951-06:00 [pool-12-thread-2] ERROR opId= com.vmware.vim.sms.provider.vasa.VersionHandler - [getVmomiServerVersion] Could not find version class for: null
2019-11-18T09:20:43.951-06:00 [pool-12-thread-2] ERROR opId= com.vmware.vim.storage.common.VmodlErrorStrings - Validation of VMOMI server version got from provider failed
com.vmware.vim.sms.fault.VasaServiceException: Could not find version class for: null
        at com.vmware.vim.sms.provider.vasa.VersionHandler.getVmomiServerVersion(VersionHandler.java:430)
        at com.vmware.vim.sms.provider.vasa.VersionHandler.validateVmomiServerVersion(VersionHandler.java:448)
        at com.vmware.vim.sms.provider.vasa.VersionHandler.retrieveVmodlProviderVersionInfo(VersionHandler.java:151)
        at com.vmware.vim.sms.provider.ProviderFactory.createVmodlVasaProvider(ProviderFactory.java:279)
        at com.vmware.vim.sms.provider.ProviderFactory.createProvider(ProviderFactory.java:168)
        at com.vmware.vim.sms.StorageManagerImpl.registerProviderInt(StorageManagerImpl.java:483)
        at sun.reflect.GeneratedMethodAccessor993.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.vmware.vim.sms.task.JobHandler.run(JobHandler.java:70)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-11-18T09:20:43.951-06:00 [pool-12-thread-2] INFO  opId= com.vmware.vim.sms.StorageManagerImpl - Timer stopped: registerProviderInt, Time taken: 14 ms.
2019-11-18T09:20:50.545-06:00 [pool-7-thread-2] INFO  opId=sps-DTCPoller-87550-487 com.vmware.pbm.qs.listener.DatastoreTagChangePoller - Starting Timer: DatastoreTagChangePoller.
 

• vmware-vsan-health-service.log:

2019-11-18T09:16:31.70Z INFO vsan-mgmt[MainThread] [VsanVmomiPatcher::PatchMoManager opID=noOpId] MoManager is patched!
2019-11-18T09:16:31.131Z CRITICAL vsan-mgmt[MainThread] [VsanMgmtServer::UncaughtExcpetionHandler opID=noOpId] Traceback (most recent call last):
   File "/usr/lib/vmware-vpx/vsan-health/VsanMgmtServer.py", line 436, in <module>
    _VsanMgmtServer.Start()
 pyVmomi.VmomiSupport.vmodl.RuntimeFault: (vmodl.RuntimeFault) {
   msg = 'N7Vmacore23FileIONotFoundExceptionE: Could not find file  : /etc/vmware-vsan-health/.cns_pgpass',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []

Environment

VMware vCenter Server Appliance 6.7.x

Cause

The issue is caused due to the .cns_pgpass file is missing under /etc/vmware-vsan-health/ caused by the update.

Resolution

In order to resolve the issue, please execute the below steps:

1. Log in to the VCSA via ssh (putty)
2. Execute the below command.

PYTHONPATH=/usr/lib/vmware-vpx/pyJack/:/usr/lib/vmware/site-packages/  python /usr/lib/vmware-vpx/firstboot/CnsVcConfigure.py
 

3. Start the service using the command:

service-control --start vmware-vsan-health

Additional Information

Impact/Risks:
VAMI based backups will fail since the service is stopped
vSAN monitoring information would not be available