Unable to Power off / Shut down the VM's from vCenter UI with multiple failed attempts

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VM hard stop task from vCenter UI is hung at 0% :

 

 

• From ESXi ssh, tasks show in "queued" status:

1. Below command shows the task list running on the ESXi level:

[root@:~] vim-cmd vimsvc/task_list
(ManagedObjectReference) [

   'vim.Task:haTask-xx-vim.VirtualMachine.shutdownGuest-xxxxxxx',
   'vim.Task:haTask-xx-vim.VirtualMachine.shutdownGuest-xxxxxxx'

     ii. ESXi task specific to the impacted vm showing tasks in "queued" status:

[root@:~] vim-cmd vimsvc/task_info haTask-59-vim.VirtualMachine.shutdownGuest-xxxxxxx
(vim.TaskInfo) {
   key = "haTask-59-vim.VirtualMachine.shutdownGuest-xxxxxxx",
   task = 'vim.Task:haTask-59-vim.VirtualMachine.shutdownGuest-xxxxxxx',
   description = (vmodl.LocalizableMessage) null,
   name = "vim.VirtualMachine.shutdownGuest",
   descriptionId = "VirtualMachine.shutdownGuest",
   entity = 'vim.VirtualMachine:xx',
   entityName = "VM-name",
   locked = <unset>,
   state = "queued",
   cancelled = false,
   cancelable = true,
   error = (vmodl.MethodFault) null,
   result = <unset>,
   progress = <unset>,
   progressDetails = <unset>,
   reason = (vim.TaskReasonUser) {
      userName = "root"
   },
   queueTime = "YYYY-MM-DDTHH:MM:SSZ",
   startTime = <unset>,
   completeTime = <unset>,
   eventChainId = xxxxxxx,
   changeTag = <unset>,
   parentTaskKey = <unset>,
   rootTaskKey = <unset>,
   activationId = <unset>
}

 

• From ESXi ssh, "df -h" command throw below error for the datastore where the vm resides:

VmFileSystem: Slow refresh failed: Unable to get FS Attrs for /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx': Unable to get FS Attrs for /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx, skipping.

 

• Logs from ESXI ssh show below errors:

1. /var/log/vmkwarning.log throw below error:

YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpuxx:xxxxxx)WARNING: Partition: xxxx: Partition table read from device naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx failed: I/O error
YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpux:xxxxxx)WARNING: Partition: xxxx: Partition table read from device naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx failed: I/O error
YYYY-MM-DDTHH:MM:SSZ Al(177) vmkalert: cpuxx:xxxxxx)ALERT: hostd performance has degraded due to high system latency

      ii. SCSI read errors (0X28) on the storage seen in /var/log/vmkernel.log:

YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpuxx:xxxxxx)ScsiDeviceIO: xxxx: Cmd(0xxxxxxxxxxxxx) 0x28, CmdSN 0xxxxxx from world 0 to dev "naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x84 0x0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpux:xxxxxx)ScsiDeviceIO: xxxx: Cmd(0xxxxxxxxxxxxxx) 0x28, CmdSN 0xxxxxx from world 0 to dev "naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x84 0x0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpuxx:xxxxxx)ScsiDeviceIO: xxxx: Cmd(0xxxxxxxxxxxxxx) 0x28, CmdSN 0xxxxxx from world 0 to dev "naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x84 0x0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpuxx:xxxxxx)NMP: nmp_ThrottleLogForDevice:xxxx: last error status from device naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx repeated 10 times

Environment

VMware ESXi 8.x

Cause

SCSI read (0X28) errors on the datastore where VM resides

Resolution

Contact storage vendor to fix the storage layer errors