vsan-health service does not start in a CNS environment

Products

• VMware vCenter Server 8.0
• VMware vSphere with Tanzu

Issue/Introduction

• Attempting to start the vsan-health fails with error "Service crashed while starting"
• Attempting to start from command line using  command "service-control --start --vsan-health" fails with below error
  Service-control failed .Error:Failed to start services in profile ALL. RC=2, stdrr=Failed to start vsan-health services .Error :Service crashed while starting 

• In vCenter /var/core you will see core.CnsSyncMgr..xxxx
• In VAMI backup you will see error "Invalid vCenter Server Status: All required services are not up! Stopped services : 'vsan-health'.
• In /var/log/vmware/vsan-health/vsanvcmgmtd-*.log you will see similar to below error
  
  yyyy-mm-ddThh:mm:ss.us: error vsanvcmgmtd[155487] [vSAN@6876 sub=WorkQueue.vmacoreDefaultLongTaskQueue] Received a negative scheduler interval: -694967296
• Prior to service not starting you have implemented incorrect value for "startupSyncInterval"  in CNS config file in vCenter
  
  /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml

Cause

startupSyncInterval setting having incorrect value crashes the vsan-health service.

Resolution

1. Verify by navigating to file /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml
     If the value is different like below
    <fullSyncInterval>600</fullSyncInterval>
         <!-- new sync interval by second, 0 means disabled -->
         <newSyncInterval>31536000</newSyncInterval>
         <!-- determine how many times to do startup fullsync-->
         <startupSyncTry>5</startupSyncTry>
         <!-- determine interval between startupSyncs-->
         <startupSyncInterval>31536000</startupSyncInterval>

The default value is like below

    <fullSyncInterval>600</fullSyncInterval>
         <!-- new sync interval by second, 0 means disabled -->
         <newSyncInterval>60</newSyncInterval>
         <!-- determine how many times to do startup fullsync-->
         <startupSyncTry>5</startupSyncTry>
         <!-- determine interval between startupSyncs-->
         <startupSyncInterval>60</startupSyncInterval>

NOTE: In versions prior to vCenter 8.0 U3  the file is located at /usr/lib/vmware-vpx/vmware-vsan

This was the result while attempting to implement the workaround as per release notes for volume sync errors  - VMware vSphere IaaS Control Plane 8.0 Release Notes - Storage Known Issues

2. Change the value to 

    <fullSyncInterval>600</fullSyncInterval>
         <!-- new sync interval by second, 0 means disabled -->
         <newSyncInterval>31536000</newSyncInterval>
         <!-- determine how many times to do startup fullsync-->
         <startupSyncTry>5</startupSyncTry>
         <!-- determine interval between startupSyncs-->
         <startupSyncInterval>60</startupSyncInterval>

3.Start the vsan-health service using command service-control --start --vsan-health