Powering off an unresponsive virtual machine on an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for shutting down an unresponsive virtual machine from the ESXi CLI when facing below scenario:

Unable to power off virtual machine on an ESXi host.
A virtual machine is unresponsive and cannot be forced off or stopped.
The host cannot access the virtual machine or unlock files.
After shutting down a virtual machine, vCenter Server shows the virtual machine as running.
There is no indication that a virtual machine is shut down.
Edit Settings option is unavailable for the virtual machine.
Attempting actions on the VMs outputs one or more of these errors:

A general system error occurred: Syscall kill returned error (-1) during vm termination attempt: ## with cartel id: ## and error message: No such process

Soap error 999. The operation is not allowed in current state

The attempted operation cannot be performed in the current state (Powered Off)

The request refers to an object that no longer exists or has never existed

Environment

VMware ESXi 6.x , 7.x , 8.x

Resolution

If powering off the virtual machine from vSphere Client or Host Client will not work, use one of the command line methods below.

Determining the virtual machine's location

Determine the host on which the virtual machine is running.
This information is available in the virtual machine's Summary tab when viewed in vCenter web client

Using the CLI to shutdown the unresponsive virtual machine

There are several methods to power off the non-responsive virtual machine using command line as the following:

It's recommended to follow the sections and steps in this article in order.

Using vim-cmd

Log in as root to the ESXi using SSH.
Get a list of all registered virtual machines, identified by their VMID and Display Name by running this command:
```
vim-cmd vmsvc/getallvms
```

Example output:

Take note of the impacted virtual machine ID VMID
Get the current state of a virtual machine by running this command:
```
vim-cmd vmsvc/power.getstate VMID
```

Example output:

Check if there hanging task on the impacted virtual machine that prevent the machine from powering on by running this command:
```
vim-cmd vmsvc/get.tasklist VMID
```

Example output:

(ManagedObjectReference) [
'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-1234567',
'vim.Task:haTask-2-vim.VirtualMachine.consolidateDisks-12345678'
]

Run the vim-cmd vimsvc/task_info task_id command to see the task in more detail.

For example using the data from step 5:

vim-cmd vimsvc/task_info haTask-2-vim.VirtualMachine.createSnapshot-1234567

vim.TaskInfo) {
dynamicType = <unset>,
key = "haTask-2-vim.VirtualMachine.createSnapshot-1234567",
task = 'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-1234567',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.createSnapshot",
descriptionId = "VirtualMachine.createSnapshot",
entity = 'vim.VirtualMachine:2',
entityName = "VM_name",
state = "running",
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 50,
reason = (vim.TaskReasonUser) {
dynamicType = <unset>,
userName = "********",
},
queueTime = "YYYY-MM-DDThh:mm:ss.msZ",
startTime = "YYYY-MM-DDThh:mm.ss.msZ",
completeTime = <unset>,
eventChainId = 1234567,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
}

The task is hung and needs to be cancelled, use the command:
```
vim-cmd vimsvc/task_cancel task_id
```

Note: Please note that not all tasks can be canceled. For instance, in the example above we see the snapshot task has the cancelable flag set to false (cancelable = false).

Shutdown the virtual machine using the VMID found in Step 3 and run this command:
```
vim-cmd vmsvc/power.shutdown VMID
```

Note: If the virtual machine fails to shut down, run this command to hard power-off the VM:

vim-cmd vmsvc/power.off VMID

Example output:

Using esxcli

Note: If esxcli does not work, use localcli instead by replacing esxcli with localcli.

Log in as root to the ESXi using SSH.
Get a list of running virtual machines, identified by World ID and Display Name by running this command:
```
esxcli vm process list
```

Example output:

Take a note of the world ID of the non-responsive virtual machine. For example in step 3 the World ID is 2750434.
Power off the virtual machine from the list by running one of these commands:
esxcli vm process kill -t soft -w WorldID
or
esxcli vm process kill -t hard -w WorldID
or
esxcli vm process kill -t force -w WorldID

Note: These command provide different levels of force to stop the virtual machine:

Soft is the most graceful
Hard performs an immediate shutdown
Force should be used as a last resort

If the VM fails to stop even after running the "esxcli vm process kill" command, evacuate all other VMs off the ESXi host and restart the ESXi itself.
If the ESXi host cannot be managed by vCenter (e.g., vMotion does not work for VMs) and the "localcli vm process kill" command returns a "No such process" error or returns no error but the VM still appears in the "localcli vm process list" output, back up the running VMs in the Guest OS if necessary, then power off and power on the ESXi server during a maintenance window.

Using esxtop

Killing a virtual machine using esxtop is equivalent to a forced hard shutdown with esxcli.

Log in as root to the ESXi host using SSH
Show a list of the currently running virtual machines with esxtop
```
esxtop
```
Press "Shift-V" to show only running virtual machines.
Press "F" to add new fields, and then "C" to select the LWID (Leader World Id) field. Press "Enter" to apply the change
To kill one of the virtual machine processes, Press "K" and copy the LWID from the unresponsive VM.

In case none of the above methods work, it might be necessary to reboot the ESXi host to get rid of the stale vm process