Virtual machine failed to power on with error "Disconnected from virtual machine. Remote connection failure Failed to establish transport connection."

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machine reported hung and guest OS is inaccessible.
• VM reset fails to power on the virtual machine.

Environment

ESXi 7.0.x
ESXi 8.0.x

Cause

Multiple luns reported read and write IO failures in the kernel logs. Failed with H:0x0 D:0x28 P:0x0 response.
This status is returned when the LUN prevents accepting SCSI commands from initiators due to lack of resources, namely the queue depth on the array. 

Sample vmkernel log :

cpu18:2098366)ScsiDeviceIO: 4115: Cmd(0x45d9623c8007) 0x2a, CmdSN 0xabc from world 2102221 to dev "naa.#############################" failed H:0x0 D:0x28 P:0x0
cpu18:2098366)ScsiDeviceIO: 4115: Cmd(0x45b964619686) 0x8a, CmdSN 0x369 from world 2102530 to dev "naa.#############################" failed H:0x0 D:0x28 P:0x0
cpu18:2098366)ScsiDeviceIO: 4115: Cmd(0x45b959b41991) 0x8a, CmdSN 0x123 from world 2102530 to dev "naa.#############################" failed H:0x0 D:0x28 P:0x0

Resolution

Contact Storage vendor to check the congestion in the SAN network and array resource utilization. Once the issue is addressed from array end, rescan /refresh the storage adapters on ESXi host.
To perform rescan follow KB Performing a rescan of the storage on an ESXi host