The VM was unexpected rebooted due to an issue with FC Storage HBA connected to a Storage Array

Products

• VMware vSphere ESXi

Issue/Introduction

Losing access to datastores.

The host UI reported that the host was powered off through the message: Alarm 'Virtual machine error' on vm_name triggered by event ######## 'Issue detected for vm_name on ESXi_hostname in cluster_name: Watchdog has fired, causing VM to reset/power off.' at DD/MM/YYYY, HH:MM:SS.

vmware.log:
YYYY-MM-DDTHH:MM:SS No(##) vmx - ConfigDB: Setting vwdt.watchdogFired = "TRUE"
YYYY-MM-DDTHH:MM:SS In(##) vmx - VWDT: Resetting VM since watchdog fired.

vmkernel.log:
YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu###:#######)qlnativefc: vmhba#(##:#.#): qlnativefcEhAbort:####:C#:T#:L#: Abort command succeeded -- #
YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:#######)NMP: nmp_ThrottleLogForDevice:####: Cmd #x## (#x##c########, #######) to dev "naa.#####ac######################b##" on path "vmhba#:C#:T#:L#" Failed:
YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:#######)NMP: nmp_ThrottleLogForDevice:####: H:#x# D:#x# P:#x# . Act:EVAL. cmdId.initiator=#x########aa## CmdSN #x#####ab
YYYY-MM-DDTHH:MM:SS Wa(###) vmkwarning: cpu#:#######)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:###: NMP device "naa.#####ac########################b##" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu##:#######)VSCSI: ####: handle ################(GID:######)(vscsi#:#):Destroying Device for world ####### (pendCom 0)

Environment

VMware vSphere ESXi 8.x

Cause

Invalid TX Word Count is higher than expected

Resolution

Engage the storage vendor to check the below:

Physical cables or SFP for issues (CRC errors on the port group and so on) and make the necessary changes as per the storage vendor recommendation.

Additional Information

This KB can be used to add virtual watchdog timer device: Add a Virtual Watchdog Timer Device to a Virtual Machine to prevent the VM from unexpectedly rebooting during the storage issue.