ESXi management, esxtop, localcli functions, WebUI and df commands do not function as expected

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi host may experience the following 

• WebUI shows the error upstream connect error or disconnect/reset before headers. reset reason: connection termination
• Command line df results in the errors 
  
  Exception ignored error evaluating path:
Traceback (most recent call last):
File "<frozen getpath>", line 353, in <module>
OSError: [Errno 5] Input/output error
Fatal Python error: error evaluating path
Python runtime state: core initialized

or

VmFileSystem: Slow refresh failed: Cannot open volume: /vmfs/volumes/<UUID>

esxtop commands take excessively long to populate results or hang indefinitely 
• localcli network ip connection list returns Errors: Unable to get connection list: Unable to Get Instance List: Failure: VSI node (117:VSI_NODE_net_tcpip_instances_connections_tep)
• ESXi host is disconnected from vCenter
  
  
  

vmkernel.log 

<DATE> In(###) vmkernel: cpu#:####)ScsiDeviceIO: ####: Task mgmt request issued to device naa.######## is stuck (WorldID ####, Cmd 0x8a, CmdSN 786ef). Issuing yellow notification to the application
<DATE> In(###) vmkernel: cpu##:####)ScsiDeviceIO: ####: Task mgmt request issued to device naa.######## is stuck (WorldID ####, Cmd 0x89, CmdSN 786eb). Issuing red notification to the application
<DATE> Wa(###) vmkwarning: cpu##:####)WARNING: ScsiDeviceIO: ####: IO stuck on device naa.########for more than 120000 seconds after TM is issued
<DATE> In(###) vmkernel: cpu##:####)PowerPath:VolProbe: device naa.######## ref Cnt 17 in on state 0 suspendIoMask 1 nOpsPending 0
<DATE> In(###) vmkernel: cpu##:####)PowerPath:VolProbe: Wait for volume naa.###### to be unquiesced

An example of a process running out of memory

<DATE> In(182) vmkernel: cpu#:####)Admission failure in path: host/vim/vmvisor/backup.sh:sh.####
<DATE> In(182) vmkernel: cpu#:####)sh.#### (###) requires 70792 KB, asked 70792 KB from backup.sh (###) which has 3068 KB occupied and 68612 KB available.
<DATE> Wa(180) vmkwarning: cpu4:####)WARNING: UserParam: ####: sh: could not SetAlloc on container(####) -- #####pages: Admission check failed for memory resource

Cause

Storage I/O is aborted, not completed from the device layer, and it never made it to the target. It was queued at the device layer in the storage stack possibly due to a full queue depth.  This can result in processes running out of memory. 

Resolution

• Engage storage vendor for cause and resolution at the device layer
• Once the storage issue is resolved vMotion running VMs if possible, otherwise cleanly shutdown guests OS
• Reboot the ESXi host