VM cannot read attached disks, citing "Input/Output" errors

Products

• VMware vSphere ESXi

Issue/Introduction

When SSH'd into a Linux-based VM, disks are viewable by running commands like df -h , but when trying to read from the disks, and input/output error is thrown.

 

Example:

user@linux-vm [ ~ ]# df -h | head -n 4
Filesystem                                   Size  Used Avail Use% Mounted on
/dev/sda3                                    488M   60M  392M  14% /boot
/dev/sda2                                     10M  2.1M  7.9M  22% /boot/efi

user@linux-vm [ ~ ]# ls -l /dev/sda3
ls: cannot open directory .: Input/output error

Cause

The VM's PVSCSI driver is experiencing queue exhaustion or driver timeouts. This occurs when the virtual machine generates I/O requests faster than the storage subsystem can process them, causing the queue to fill up and requests to time out.

Resolution

   A - The VM needs to be troubleshooted and reviewed. Either I/O needs to be reduced, or the queue depth used by the PVSCSI adapter needs to be increased, allowing more I/O messages to be processed. If necessary, consider using mutliple PVSCSI controllers to load balance the I/O.

   B - Troubleshoot the driver used by the VM for I/O processing.