ESXi Host Experiences Storage Performance Degradation, Path Instability, and Asynchronous UNMAP Failures on VMFS6 Datastores
Article ID: 441677
Updated On:
Products
Issue/Introduction
An ESXi host experiences severe storage performance degradation, path instability, and space reclamation (UNMAP) failures on a VMFS6 volume.
Symptoms:
- The storage subsystem systematically cancels pending SCSI WRITE operations (Opcode
0x2a) via driver-initiated host aborts, causing Native Multipathing (NMP) to flag target paths as suspect: - You may see the below information in
/var/run/log/vmkernel.log
lpfc: lpfc_handle_status:5631: <vmhba#> FCP cmd x2a failed <0/103> sid x141600, did x140100, oxid x283e iotag x1364 Abort Requested Host Abort Req NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a to dev "<NAA_ID>" on path "<vmhba#>" Failed: H:0x5 D:0x0 P:0x0. Act:EVAL WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "<NAA_ID>" state in doubt; requested fast path state update. ScsiDeviceIO: 4656: Cmd(0x2a), failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer.
- The affected VMFS6 volume records continuous transaction log failures during block space reclamation routines:
Res6: 2944: '<DATASTORE_NAME>': RC Lock not free for type 1, return TXN FULL Fil6: 3797: <DATASTORE_NAME>: <FD c22 r22> - Failed to unmap file blocks 0/1: Transaction ran out of lock space or log space
- These symptoms may be isolated to a single host while adjacent cluster nodes accessing the same shared volume function normally without recording metadata transaction log exhaustion (
TXN FULL).
Environment
VMware ESXi 8.x
Cause
The root cause is a driver-firmware mismatch on the Emulex Fibre Channel adapters. In this instance for example, the host was running HBA firmware 14.2.589.21 with a newer 14.4.x driver branch.
This mismatch causes the driver to incorrectly handle command completion timeouts for SCSI WRITE (0x2a) operations. The resulting "Host Aborts" (DID_ABORT) prevent the VMFS6 resource control layer from completing metadata updates. This leads to a backlog of "stuck" locks that eventually exhaust the hidden transaction log space (TXN FULL).
Resolution
1. Update the Emulex HBA firmware to a version certified for use with the installed driver per the Broadcom | VMware | Hardware Compatibility Guide
2. Engage your storage vendor to inspect the physical fabric for:
- CRC errors on the switch ports connected to the affected host identifiers.
- Faulty SFPs or cables on vmhba's
- Storage controller saturation/latency logs during the timeframe of the aborts.
Additional Information
Feedback
