VMs on vSAN fail to power on with Unable to enumerate all disks and vmdks report 0MB in size

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• VMs reside on vSAN datastore
• All vSAN objects are reporting as healthy
• vmdks show 0Mb in size
• VMs fail to power on with the below error

You see the below messages in the vmware.log:
2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/07c39b5c-50da-2693-29fd-e4434b4bfc90/GBMSUS1616SVR14.vmdk
2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/07c39b5c-50da-2693-29fd-e4434b4bfc90/GBMSUS1616SVR14.vmdk
2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/07c39b5c-50da-2693-29fd-e4434b4bfc90/GBMSUS1616SVR14.vmdk
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 7.0.x
VMware vSAN 8.0.x
VMware vSAN 6.x

Cause

The vmdk backing object contains a different vSAN datastore UUID in the description path.

To confirm this do the following:

[root@MSUS1616ESX003:/vmfs/volumes/vsan:521a060e917bb08e-560a1571aa386106/07c39b5c-50da-2693-29fd-e4434b4bfc90] cat GBMSUS1616SVR14.vmdk | grep RW
RW 125829120 VMFS "vsan://09c39b5c-eeb3-4b65-9737-e4434b4bfc90"

[root@MSUS1616ESX003:/vmfs/volumes/vsan:521a060e917bb08e-560a1571aa386106/07c39b5c-50da-2693-29fd-e4434b4bfc90] localcli vsan debug object list -u 09c39b5c-eeb3-4b65-9737-e4434b4bfc90|grep Path:
Path: /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/07c39b5c-50da-2693-29fd-e4434b4bfc90/GBMSUS1616SVR14.vmdk (Missing)

As you can see from the highlighted sections the UUIDs don't match.

Resolution

Perform the following steps to resolve this issue.

1) Run the below command to get a list of all affected objects after navigating to the vSAN datastore
   [root@MSUS1616ESX003:/vmfs/volumes/vsan:521a060e917bb08e-560a1571aa386106] localcli vsan debug object list|grep -A1 (Missing)|less
Note: For large environments you may want to send it to a file by replacing |less with > /tmp/<filename>.txt   
   Sample output:
    /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/MSUS1616WFS001 (Missing)
    Group UUID: 24e99b5c-6839-451e-357d-e4434b4bfc90
   /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/5dc79b5c-de3d-f7b7-8714-e4434b4bf1cc/GBMSUS1616SVR18.vmdk (Missing)
  /vmfs/volumes/vsan:52e929264dfd46ad-dcf678945db0f430/GBMSUS1616SVR17 (Missing)
 Group UUID: 97c69b5c-ead9-0a37-c7da-e4434b4bf1cc
 Note: The Group UUID coresponds to the namespace UUID which is the friendly name of the VM folder 
     
2) Open a case with VMware support for assistance with correcting the vSAN UUID mismatch.

Additional Information

Impact/Risks:
VMs fail to power on.