VM Guest OS may show "Input/output error" when attempting to access local mount points

Products

• VMware vSphere ESXi

Issue/Introduction

• Guest operating systems may fail to access mounted volumes after a temporary storage connectivity disruption, such as a storage array controller failover.
• Despite redundant multipathing being active and available, the underlying operating system fails to automatically recover the mount state. 

• This directly results in application downtime for any virtual machines dependent on those storage volumes.

Environment

VMware vSphere ESXi 8.x

Cause

A preceding storage array controller failure caused a transient loss of access to the backing disks. While the ESXi host may have successfully executed a path failover to the secondary storage controller, the Guest OS (Worker Node) sustained a synchronous I/O timeout during the interruption. To prevent data corruption, the Linux kernel subsequently marks the filesystem as stale or inconsistent, directly generating the persistent "Input/output error" observed at the command-line interface.

Resolution

Run the following checks to rule out vSphere level issues:

• Verify that the backing LUN device is accessible, the associated datastore is actively mounted, and all resident VMDK files are present and intact.
• Verify within the virtual machine's "Edit Settings" menu that all associated virtual disks are present, attached, and accessible.
  VM --> Right click "Edit Settings" --> Check all individual disks
• Verify that all virtual disks attached to the virtual machines are reporting their correct provisioned capacity.
• Run "ESXTOP" and evaluate the DAVG/cmd and KAVG/cmd metrics to confirm that storage latency values align with the expected performance baselines for the specific workload and underlying storage architecture.

 

If the above checks are successful, engage the Guest OS vendor for further support.