VMs on a single host experience extreme storage latency
search cancel

VMs on a single host experience extreme storage latency

book

Article ID: 407368

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

VMs on a single ESXi host experience extreme storage read and write latency which clears once they are migrated to another host attached to the same storage.

Environment

ESXi (all versions)

Cause

The connection between the host and the storage array is unstable across one path.
An unstable connection to the storage array from the host causes frequent retransmissions of data from both the host and storage sides which increases latency as more data must be retransmitted.
This can be verified by path error, HBA busy, and HBA abort messages in the logs:

2025-08-13T14:50:25.237Z In(182) vmkernel: cpu25:2097823)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a (0x45d9465b49c0, 6106960) to dev "naa.600################" on path "vmhba2:C0:T0:L0" Failed:
2025-08-13T14:50:25.237Z In(182) vmkernel: cpu25:2097823)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4306cfd4a280 CmdSN 0x712978
2025-08-13T14:50:25.237Z Wa(180) vmkwarning: cpu25:2097823)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600################" state in doubt; requested fast path state update...
2025-08-13T14:50:43.168Z In(182) vmkernel: cpu0:2097547)lpfc: lpfc_handle_status:5631: vmhba2 3271: FCP cmd xf1 failed <0/1> sid x000002, did x000001, oxid x12b iotag x451 Abort Requested Host Abort Req
2025-08-13T14:50:43.169Z In(182) vmkernel: cpu2:2097820)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0xf1 (0x45b935bf7b40, 2097192) to dev "naa.600################" on path "vmhba2:C0:T0:L1" Failed:
2025-08-13T14:50:43.169Z In(182) vmkernel: cpu2:2097820)NMP: nmp_ThrottleLogForDevice:3898: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4306cfd5a340 CmdSN 0x1acd7c
2025-08-13T14:50:51.457Z In(182) vmkernel: cpu27:2097237)ScsiDeviceIO: 13380: Task mgmt request issued to device naa.600################ is stuck (WorldID 2097192, Cmd 0xfe, CmdSN 1acd7c). Issuing yellow notification to the application

Resolution

  • Ensure all drivers and firmware are as per the Broadcom HCL to ensure the HBA is able to perform correctly.
  • Work with the physical SAN team to troubleshoot connectivity between the host HBA and storage array to identify any failing components.

Additional Information

Powered by Wolken Software