ESXi hosts report "closed with dirty buffers" and failed ATS locking (Sense Key 0x4) due to storage hardware error

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts report multiple storage-related alarms and performance degradation.

• Specific symptoms include:
  
  
  • vCenter Alarms: Issue detected on <ESXi host name> in ha-datacenter: BC: 3160: File host-##-hb closed with dirty buffers. Possible data loss.
    
    The host-##-hb is a VMFS Heartbeat file. Each ESXi host that uses a shared VMFS datastore is assigned a unique 512-byte "heartbeat slot" within this file to indicate its liveness and manage on-disk locks.
    
    
  • Host Connectivity: VMFS datastores may report 0 bytes capacity in the vSphere Client or Host Client.
    
    
  • Command Latency: Commands like esxcli storage filesystem list may hang or fail to return a prompt.

 

• In the vmkernel.log, failed COMPARE AND WRITE (SCSI opcode 0x89) commands with a HARDWARE ERROR sense key may be seen:

ScsiDeviceIO: 4325: Cmd(0x...) 0x89, ... failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0

• • H:0x0: Host Status OK
  • D:0x2: Check Condition
  • 0x4 0x0 0x0: Sense Key 0x4 (HARDWARE ERROR), ASC 0x0, ASCQ 0x0 (NO ADDITIONAL SENSE INFORMATION)

For details on each code, refer to the KB Interpreting SCSI sense codes in VMware ESXi

Cause

The issue is caused by a physical hardware failure on the storage array. This results in the storage controller returning a specific SCSI error code to the ESXi host during Atomic Test and Set (ATS) operations.

Resolution

Because the error is explicitly reported by the storage hardware as a HARDWARE ERROR (Sense Key 0x4), the resolution must be performed at the storage layer:

1. Identify the affected device: Note the device ID (naa.<id>) from the failed SCSI command logs.
2. Contact Storage Vendor: Provide the hardware error codes and device IDs to the storage vendor for hardware diagnostics.
3. Repair/Replace Hardware: The vendor must address the underlying physical issue (e.g., failed controller, cache module, or disk).
4. Verification: Once the hardware issue is fixed, verify that the vmkernel.log no longer reports HARDWARE ERRORs and that datastore capacity and file access are restored.