Oracle RAC Node Eviction on ESXi due to qlnativefc SCSI Aborts

Products

• VMware vSphere ESXi

Issue/Introduction

• An Oracle RAC cluster node running on a VMware ESXi host experiences a sudden reboot or node eviction.
• Guest OS (RHEL/Oracle Linux) logs report high I/O wait times and SCSI task aborts: kernel: sd 4:0:1:0: [sdg] tag#678 task abort on host 4
• Oracle alert logs report ORA-29770, indicating the Log Writer (LGWR) process has been blocked (typically for >70 seconds).
• The ESXi vmkernel.log shows SCSI aborts and timeouts for the corresponding device: WARNING: NMP: nmp_ThrottleLogForDevice:3868: H:0x8 D:0x0 P:0x0 . Act:EVAL. qlnativefcEhAbort:2745:SCSI command timeout counter incremented to 4

Environment

ESXi 8.x

Oracle RAC

Cause

The issue is caused by transient storage unresponsiveness or SAN fabric latency.

In this scenario, a critical database I/O (such as a WRITE(10) from the Oracle LGWR process) is issued to the storage array but is not acknowledged. Because the Oracle RAC eviction threshold is typically shorter (~70 seconds) than the default ESXi storage driver timeout for Task Management Aborts (often 120 seconds for qlnativefc), the database cluster initiates a node eviction to protect data integrity before the hypervisor or HBA driver can recover the stuck I/O queue.

The presence of Error Handling (EH) Abort increments in the HBA driver logs without a subsequent LUN or Virtual Reset indicates that the HBA successfully cleared individual commands, but the duration of the stall was sufficient to crash the application layer.

Resolution

Because the root cause resides in the physical storage layer or fabric, investigate the following areas:

1. Analyze SAN Fabric Health
   Check Fibre Channel switch logs for evidence of dropped frames, CRC errors, or port resets on the specific HBA port (e.g., vmhba3) identified in the ESXi logs.
   Verify if the issue is isolated to a single path or HBA, which may indicate a faulty SFP, cable, or HBA port.
2. Review Storage Array Performance
   Correlate the exact timestamp of the ESXi H:0x8 (Timeout) errors with the storage array’s internal performance logs.
   Look for controller failovers, back-end disk latency spikes, or LUN-level congestion on the affected device (naa.ID).
3. Engage OS team for VM optimization best practices
4. Verify Driver and Firmware Compatibility
   Confirm the HBA driver (qlnativefc) and firmware versions are compliant with the Broadcom Compatibility Guide