vSAN VM Power On Failure - Storage Policy Change failure: 5 (Input/output error) - Stuck Object Resync

Products

• VMware vSAN
• VMware Cloud Foundation

Issue/Introduction

• A VM on a vSAN datastore fails to power on with errors:
  • "File was not found. The system cannot find the file specified. Unable to enumerate all disks." 
  • "Storage Policy Change failure: 5 (Input/output error)."
    
    
    
• The host.log on the ESXi host shows similarly:
  
  [/var/log/hostd.log on ESXi host]
  
  

  2024-06-23T17:26:55.835Z Er(163) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] OBJLIB-VSANOBJ: VsanObjReadPolicyInt: Failed to readPolicy for object ####-####-####-#########: Input/output error (327682).
  2024-06-23T17:26:55.835Z Er(163) Hostd[2103480]: [Originator@6876 sub=Libs opID=lxlcjff5-159689-auto-3f7u-h5:70061404-98-01-01-c4-3eb4 sid=52acff62 user=vpxuser:EXAMPLE\user] OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://#####-#####/####-####-####-#########'.

• The Cluster -> Monitor -> vSAN -> Resyncing Objects page shows that the VM is resyncing, but never completes.
• vSAN Skyline Health is reporting no issues
• All vSAN objects are reporting as healthy in Cluster -> Monitor -> vSAN -> Virtual Objects

Environment

• VMware vSAN 7.x
• VMware Cloud Foundation (VCF)

Cause

• The Cluster Monitoring, Membership, and Directory Service (CMMDS) database contains a stale entry that is preventing the object from properly instantiating on a particular host. As a result, one or more of the virtual disks is inaccessible. 

Resolution

1. Open a case for Broadcom Technical support (Creating and managing Broadcom cases) for further assistance to determine if the object can be recovered.
   
   
2. Otherwise, the VM will need to be restored from a backup. 

Additional Information

• NOTE: If the object is not found in esxcli vsan debug object list and is not stuck resyncing, see KB: VMs on vSAN fail to power on with error: Unable to enumerate all disks. 5 (Input/output error).