Event ID: 153 raised on Windows Guest OS indicating disk I/O timeout

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple "Event ID: 153" raised on a Windows Guest OS:
  • Log Name: SystemSource: disk Event ID: 153 Level: Warning
  • Description: The IO operation at logical block address 123456 for Disk 2 was retried.
  • Windows System Event Log reports:
    • Log Name: System
    • Source: disk
    • Event ID: 153
    • Level: Warning
    • Description: The IO operation at logical block address #### for Disk # was retried.
      
      
• Virtual Machine may become unresponsive or experience "Blue Screen" crashes (BSOD).
• vSphere HA may trigger a VM reset due to VMware Tools heartbeat failure.
• Performance degradation might be observed on the VMs or the Microsoft applications running on top.

Environment

VMware vSphere ESXi

Cause

When an application generates heavy I/O or the underlying storage path degrades, the Windows Guest OS may log numerous Event ID: 153 entries. This indicates that a request to a specific disk was retried because the storage subsystem did not respond within the expected timeout period.

This Event ID: 153 will usually be triggered due to:

1. Physical Layer Failure: Broken fiber cables, faulty SFPs, or failing HBA ports (asymmetric path failure).
2. SAN Fabric Issues: High CRC error counts on Fibre Channel interfaces.
3. Storage Congestion: Massive I/O bursts saturating storage queues.

Resolution

Identify if the failure is systemic or isolated to a specific path/adapter.

1.  Review ESXi Host Logs:
   • Check /var/log/vmkernel.log for storage timeout or connectivity errors:
     • Timeout Signature:

       cpu##:####)Fil6: 4094: 'DATASTORE_NAME': Fil6 file IO () : Timeout    

     • Connection Loss Signature:

       path "vmhba#:C#:T#:L#" Failed: H:0x1 D:0x0 P:0x0 (H:0x1 indicates HOST_NO_CONNECT).

2. Check Fibre Channel Statistics
   
   • Run the following command to check for hardware-level errors on the HBAs: localcli storage san fc stats get
   • Review the output for increasing counters in these fields:
     • Invalid CRC Count: Indicates SAN fabric or GBIC/SFP issues.
     • Loss of Signal Count / Link Failure Count: Indicates physical cable or port disconnects.Here is an example of the output showing "Invalid CRC Count:
       

       "FcStat:   Adapter: vmhba2   Tx Frames: 1549904085   Rx Frames: 2846844196   Lip Count: 3   Error Frames: 0   Dumped Frames: 0   Link Failure Count: 0   Loss of Signal Count: 2   PrimSeq Protocol Err Count: 0   Invalid Tx Word Count: 608297   Invalid CRC Count: 5201   Input Requests: 57621286   Output Requests: 244055658   Control Requests: 0

3. Physical Infrastructure Remediation
   
   • If errors are isolated to a single vmhba (asymmetric failure):
     1. Replace Physical Fiber Patch Cable: Swap the fiber run originating from the impacted host HBA port.
     2. Audit Switch-Side Port: Inspect the physical FC switchport. Check for a blown SFP transceiver or a disabled port.
     3. Replace HBA: If the link remains down after cable and switch-side validation, replace the physical HBA card.
        
        
4. Optimize Guest OS Settings
   • If no hardware errors are found, consider increasing the disk timeout settings within the Windows Guest OS to better handle transient I/O bursts.

Additional Information

If you believe you have encountered this issue and are unable to upgrade, open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.