Volumes could be missing from CNS DB due to corrupted metadata in all vSphere/ESX 6.7 versions prior to 6.7 P05 and all vSphere/ESX 7.0 versions prior to 7.0 U2 MP1

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Some volumes could be missing from the container volumes page, regardless if they have been created successfully. 
• CSI Syncer will observe some volumes are not present in the CNS DB, but still in use by k8s.
• In /var/log/vmware/vsan-health/vsanvcmgmtd.log, you see error message:

 

2021-06-09T04:28:56.157Z error vsanvcmgmtd[60296] [vSAN@6876 sub=CnsObjBuilder] CNS: GetEntityTypeStrFromVmodlToMeta. Invalid EntityType. type=
2021-06-09T04:28:56.159Z info vsanvcmgmtd[60296] [vSAN@6876 sub=CnsSyncMgr] CNS: [1394] Finish Sync(BLOCK) = Failure
--> Input Volumes = 769
--> insertCnt = 0
--> updateCnt = 0
--> skipCnt = 768
--> denyCnt = 0
--> deleteCnt = 0
--> updateEntityCnt = 0
--> deleteEntityCnt = 0
--> markForDeleteCnt = 0
--> finalVolumeCount = 3053
--> duration = 254204
-->
2021-06-09T04:28:56.159Z error vsanvcmgmtd[60296] [vSAN@6876 sub=CnsSyncMgr] CNS: [1394] Exception occurs during sync. CNS: GetEntityTypeStrFromVmodlToMeta. Invalid EntityType. type=

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 7.0.x
VMware vSAN 6.7.x

Cause

CNS fullSync can be interrupted by corrupted volume metadata, and cause some volumes missing from CNS DB. Volume metadata is stored in ESX host, it could be corrupted by concurrent update operations, host restarting, etc.

Resolution

This is a known issue affecting all vSphere/ESX 6.7 versions earlier than 6.7 P05, and all vSphere/ESX 7.0 versions earlier than 7.0 U2 MP1. 

The fix is available in vSphere/ESX 6.7 P05 or later, and vSphere/ESX 7.0 U2 MP1 or later.

VMware recommends upgrading to a fixed release to avoid hitting this issue.