Virtual machines power on failure with (Input/output error) on vSphere Virtual Volumes (vVol)

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Virtual machines residing on a vSphere Virtual Volumes (vVol) datastore fail to power on, resulting in an inability to access the guest operating system, applications, or data.
  
  
• Snapshot chain consistency checks pass, and the associated .vmdk files are present and report no structural issues.
  
  
  • Refer: Verifying a snapshot chain and cloning a Virtual Disk from snapshots

• The following error is observed during the power-on attempt in the vCenter UI: 
  
  (Input/output error) Failed to start the virtual machine. Cannot open the disk '/vmfs/volumes/vvol:################-################/#######.########-####-####-####-############/VM_NAME_#-######.vmdk' or one of the snapshot disks it depends on. Cannot open the disk '/vmfs/volumes/vvol:################--################/#######.########-####-####-####-############/VM_NAME_#-######.vmdk' or one of the snapshot disks it depends on. Module Disk power on failed. Cannot open the disk '/vmfs/volumes/vvol:################--################/#######.########-####-####-####-############/VM_NAME_#-######.vmdk' or one of the snapshot disks it depends on.

Environment

VMware vSphere ESXi 7.x
VMware vCenter Server 7.x
VMware vSphere ESXi 8.x
VMware vCenter Server 8.x

Cause

• The issue is caused by intermittent communication disruptions between the ESXi host and the storage array's VASA Provider or Protocol Endpoint (PE) for virtual machines residing on vSphere Virtual Volumes (vVols) datastores.
• These connectivity failures prevent the host from successfully binding to and accessing the vVol objects, which leads to I/O timeouts and virtual machine power-on failures.

Resolution

1. Permanent Resolution: Engage the storage vendor to investigate VASA Provider connectivity and apply recommended microcode patches.
2. Workaround (Interim):
   1. Identify a healthy datastore.
   2. Perform a storage vMotion of the Virtual Machine files.
   3. Power on the Virtual Machine.