Applications running on VMs on Fibre Channel storage become unresponsive

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• Certain application on certain virtual machines are reporting to be unresponsive.
• The storage in use is Fibre Channel.
• During the time of the event, aborts similar to the following are observed in ESXi /var/log/vmkernel.log (logging may vary depending on the particular vmhba driver)
  

  2025-05-22T10:41:31.189Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3345:qlnativefcEhVirtualReset: aborting sp ############## handle 67d from RISC. serialNumber=################, Command timeout=57391 sec
  2025-05-22T10:41:31.190Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3357:qlnativefcEhVirtualReset: abortCommand mbx success.
  2025-05-22T10:41:31.390Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3385:C0:T1:L216: Virtual Abort succeeded -- ####### (1)
  2025-05-22T10:41:31.390Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3316:C0:T0:L216: VIRTUAL RESET ISSUED.
  2025-05-22T10:41:31.390Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3385:C0:T0:L216: Virtual Abort succeeded -- ####### (0)
  2025-05-22T10:41:55.549Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3316:C0:T1:L216: VIRTUAL RESET ISSUED.
  2025-05-22T10:41:55.549Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3340:Command aborted on target=0x02x, lun=0xd8 - SCSI command timeout counter incremented to 4876
  2025-05-22T10:41:55.549Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3345:qlnativefcEhVirtualReset: aborting sp ############## handle 393 from RISC. serialNumber=########, Command timeout=57368 sec
  2025-05-22T10:41:55.549Z cpu21:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3357:qlnativefcEhVirtualReset: abortCommand mbx success.
  2025-05-22T10:41:55.750Z cpu20:2097452)qlnativefc: vmhba#(3b:0.0): qlnativefcEhVirtualReset:3385:C0:T1:L216: Virtual Abort succeeded -- ####### (1)

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

The logs indicate that aborts are the result of command timeouts. I/O is being issued by the driver but is failing to complete, or the driver is not receiving notification of I/O completion.

Resolution

• Verify that the vmhba driver/firmware is supported as per the Broadcom HCL - Broadcom HCL
• Investigate the fabric and SAN layers to determine the cause of the command timeouts with the fabric and SAN vendors as necessary.