VMs on vSAN fail to power on with Unable to enumerate all disks and vmdks report 0MB in size

Products

• VMware vSAN

Issue/Introduction

• VMs reside on vSAN datastore

• All vSAN objects are reporting as healthy

• vmdks show 0Mb in size

• VMs fail to power on with the error "Unable to enumerate all disks"

• You see the below messages in the vmware.log:

  2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-################/07c39b5c-50da-2693-29fd-############/<vmname>.vmdk

  2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-################/07c39b5c-50da-2693-29fd-############/<vmname>.vmdk

  2021-04-23T13:19:51.145Z| vmx| I125: OBJLIB-VSANOBJ:Failed to resolve the path name /vmfs/volumes/vsan:52e929264dfd46ad-################/07c39b5c-50da-2693-29fd-############/<vmname>.vmdk

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

The vmdk backing object contains a different vSAN datastore UUID in the description path. In vSAN, all objects have extended attributes embedded in the object metadata. One of the extended attributes is the path to the object's descriptor file.

These extended attributes (including descriptor path) are validated when the VMDK is opened. If the extended attribute path embedded in the object metadata does not match the actual path to the descriptor file, VMDK initialization will fail and the virtual machine will fail to power on with error "Unable to enumerate all disks" message.

To confirm this do the following:

Navigate to the VM folder with a cd (to vSAN datastore, then VM; this will automatically go into the VMs namespace folder). 

• cd /vmfs/volumes/<vSAN:DatastoreUUID>/<VM-name>/

Read the VMDK descriptor file: 

• cat <vmname>.vmdk | grep RW

[root@Host:/vmfs/volumes/vsan:521a060e917bb08e-################/07c39b5c-50da-2693-29fd-############] cat <vmname>.vmdk | grep RW
RW 125829120 VMFS "vsan://09c39b5c-eeb3-4b65-9737-############"

List out the path information for the vSAN object: 

• localcli vsan debug object list -u 09c39b5c-eeb3-4b65-9737-############# | grep Path:

[root@Host:/vmfs/volumes/vsan:521a060e917bb08e-################/07c39b5c-50da-2693-29fd-############] localcli vsan debug object list -u 09c39b5c-eeb3-4b65-9737-############ | grep Path:
Path: /vmfs/volumes/vsan:52e929264dfd46ad-################/07c39b5c-50da-2693-29fd-############/<vmname>.vmdk (Missing)

As you can see from the highlighted sections the UUIDs don't match.

• Current vSAN UUID: 521a060e917bb08e-################
• Metadata vSAN UUID: 52e929264dfd46ad-################


Resolution

Perform the following steps to resolve this issue.

1) Run the below command to get a list of all affected objects after navigating to the vSAN datastore

      [root@Host:/vmfs/volumes/vsan:521a060e917bb08e-################] localcli vsan debug object list|grep -A1 (Missing)|less

Note: For large environments you may want to send it to a file by replacing "|less" with "> /tmp/<filename>.txt"  

 

Sample output:

/vmfs/volumes/vsan:52e929264dfd46ad-################/VM1 (Missing)

Group UUID: 24e99b5c-6839-451e-357d-############

/vmfs/volumes/vsan:52e929264dfd46ad-################/5dc79b5c-de3d-f7b7-8714-############/TEST.vmdk (Missing)
/vmfs/volumes/vsan:52e929264dfd46ad-################/VM2 (Missing)
Group UUID: 97c69xxx-xxxx-xxxx-c7da-############

Note: The Group UUID corresponds to the namespace UUID which is the friendly name of the VM folder 
     
2) Collect logs from the entire cluster

3) Open a case with VMware by Broadcom support and upload the file generated in Step 1 along with the cluster logs for assistance with correcting the vSAN UUID mismatch.