During application Maintenace some service restart inside guest OS is delayed

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Some application services take long time to come up
• The issue is limited to VMs on a particular datastore and is no longer observed once the VMs are migrated to a different datastore.

 

Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

This occurs when the Fibre Channel HBA driver aborts I/O commands as a result of errors or timeouts in communication between the host and the storage device (LUN).

• In the /var/log/vmkernel.log file of the ESXi host following abort messages is reported by the qlnativefc driver:

YYYY-MM-DDThh:mm:ss.fff cpu94:9043914)qlnativefc: vmhba1(3a:0.1): qlnativefcEhAbort:2763:qlnativefcEhAbort: abortCommand mbx success.
YYYY-MM-DDThh:mm:ss.fff cpu74:9217137)qlnativefc: vmhba1(3a:0.1): qlnativefcStatusEntry:2077:C0:T25:L108 - FCP command status: 0x5-0x0 (0x8) portid=3e0fc1 oxid=0x31e cdb=8a0000 len=1024 rspInfo=0x0 resid=0x0 fwResid=0x0 host status = 0x8 device status $
YYYY-MM-DDThh:mm:ss.fff cpu85:2098576)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x8a (0x45ea8b350588, 9043873) to dev "naa.############################" on path "vmhba1:C0:T25:L108" Failed:
YYYY-MM-DDThh:mm:ss.fff cpu85:2098576)NMP: nmp_ThrottleLogForDevice:3875: H:0x8 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430e9e74d6c0 CmdSN 0x800e0058
YYYY-MM-DDThh:mm:ss.fff cpu85:2098576)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.############################" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.fff cpu85:2098576)ScsiDeviceIO: 4115: Cmd(0x45ea8b350588) 0x8a, CmdSN 0x800e0058 from world 9043873 to dev "naa.############################" failed H:0x8 D:0x0 P:0x0

• "qlnativefcEhAbort" typically relates to the QLogic native Fibre Channel driver in ESXi environments. This function or operation is invoked when there is a need to abort I/O commands due to errors or timeouts in communication between the host and the storage device. 
• This indicates that the driver is attempting to abort one or more outstanding I/O commands on a specific Fibre Channel (FC) path or device.

 

Resolution

To resolve this issue, involve the storage vendor to investigate potential timeouts or communication problems between the host and the storage device.

Additional Information

Japanese Version:
アプリケーションメンテナンスの際にゲスト OS の一部のサービスの再起動が遅延する