VM will not shutdown until reboot of the host

Products

• VMware vSphere ESXi

Issue/Introduction

• Issuing commands to ESXi host via UI or CLI will not finish
• In the /var/log/vmkernel.log you will see entries similar to these:
  

  qlnativefc: vmhba#(##:#.#): qlnativefcStatusEntry:1921:(1:5) Dropped frame(s) detected (53248 of 262144 bytes).
  qlnativefc: vmhba#(##:#.#): qlnativefcStatusEntry:2069:C#:T#:L# - FCP command status: 0x15-0x0 (0x2) portid=###### oxid=0x### cdb=####### len=###### rspInfo=0x0 resid=0x0 fwResid=0x#### host status = 0x2 device statu$
  NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x88 (0x############, ########) to dev "naa.####################" on path "vmhba#:C#:T#:L#" Failed:
  NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x############ CmdSN 0x########
  WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.####################" state in doubt; requested fast path state update...
  ScsiDeviceIO: 4633: Cmd(0x############) 0x88, CmdSN 0x######## from world ######## to dev "naa.####################" failed H:0x2 D:0x0 P:0x0
  

Environment

7.X

8.X

Cause

This is the result of frames getting dropped between vmhba to the storage array. 

Resolution

There appears to be an issue between the vmhba and the storage array. We recommend that the customer and the storage vendor investigate together to identify where the frame drop is occurring.