VMs on vSAN fail to power on with error: Unable to enumerate all disks. 5 (Input/output error)

Products

• VMware vSAN

Issue/Introduction

• VMs reside on vSAN datastore
• All vSAN objects are reporting as healthy
• vSAN Skyline Health is reporting no issues
• VMs fail to power on with the below error
• "File was not found. The system cannot find the file specified. Unable to enumerate all disks." error may be seen in vCenter Web Client
• ESXi - /var/log/hostd.log
  

  2024-06-23T17:26:55.835Z Er(163) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] OBJLIB-VSANOBJ: VsanObjReadPolicyInt: Failed to readPolicy for object ####-####-####-#########: Input/output error (327682).
2024-06-23T17:26:55.835Z Er(163) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://#####-#####/####-####-####-#########'.

  2024-06-23T17:26:55.835Z In(166) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] SNAPSHOT: Snapshot DiskTreeAddDiskHierarchy: Couldn't add disk '/vmfs/volumes/vsan:##/####-####-####-#########/VM.vmdk': Input/output error (5).
2024-06-23T17:26:55.836Z In(166) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] VigorOfflineGetAllDisks: Failed to retrieve disk files: Input/output error

The following commands do not return any results:

• esxcli vsan debug object list -u <OBJECT UUID>
• cmmds-tool find -f python -u <OBJECT UUID>

Cause

This is due to the backing object no longer existing on the vSAN datastore. This is typically due to a vSAN object being marked for deletion due to some sort of manual intervention, and once the locks were released (VM power-off, etc.,) on the object - vSAN proceeded with deleting the object.

vSAN doesn't just delete objects at random. If an object is deleted, it will be due to user intervention or in rare circumstances a bug in the code. VMware by Broadcom has seen these types of occurrences in the following scenarios:

• Interacting with VMware Support Log bundles within ESXi - See KB Interacting with VMware Support Log bundles within ESXi may impact vSAN Object Accessibility for more details
• User run script deleting vSAN objects (VMware by Broadcom only recommends using scripts with extreme caution to ensure what is being deleted is no longer needed)
• User accidentally deleting the wrong files/VMs from the environment 
  Note: In the above scenarios the objects were marked for deletion by vSAN however since the objects were locked due to an active process the objects were not deleted until such time that lock was released by either a VM migration, power cycle, or HA event.

Resolution

Restore the VM/affected vmdk from backup.

In order to potentially determine the cause of why the object no longer exists on vSAN, the entire cluster logs including vCenter will need to be collected as close to the time of the event as possible including the names of the affected VMs, timestamps, and time zone where the cluster is located.

These logs will need to be uploaded to VMware by Broadcom to the case opened with support for review. We will provide a best effort review of the logs provided, but can not guarantee if a cause can be found. As the object marked for deletion could have happened days, weeks, months ago and any pertinent data would no longer be available and have rolled off the system.