Powering off an unresponsive virtual machine on an ESXi host
search cancel

Powering off an unresponsive virtual machine on an ESXi host

book

Article ID: 308457

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for shutting down an unresponsive virtual machine from the ESXi CLI.

Symptoms:

  • Unable to power off virtual machine on an ESXi host.
  • A virtual machine is unresponsive and cannot be forced off or stopped.
  • The host cannot access the virtual machine or unlock files.
  • After shutting down a virtual machine, vCenter Server shows the virtual machine as running.
  • There is no indication that a virtual machine is shut down.
  • Edit Settings option is unavailable for the virtual machine.
  • Attempting actions on the VMs outputs one or more of these errors:
    • Soap error 999. The operation is not allowed in current state
    • The attempted operation cannot be performed in the current state (Powered Off)
    • The request refers to an object that no longer exists or has never existed



Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

 

Resolution

If powering off the virtual machine from vSphere Client or Host Client will not work, you can use one of the command line methods below.
 

 

Determining the virtual machine's location

  • Determine the host on which the virtual machine is running.
  • This information is available in the virtual machine's Summary tab when viewed in vCenter web client

 

Using the CLI to shutdown the unresponsive virtual machine

There are several methods to power off the non-responsive virtual machine using command line as the following:

Its recommended to follow the sections and steps in this article in order.
 


Using vim-cmd

  1. Log in as root to the ESXi using SSH.
  2. Get a list of all registered virtual machines, identified by their VMID and Display Name by running this command:
    vim-cmd vmsvc/getallvms

Example output:

  1. Take note of the impacted virtual machine ID VMID
  2. Get the current state of a virtual machine by running this command:
    vim-cmd vmsvc/power.getstate VMID
Example output:
  1. Check if there hanging task on the impacted virtual machine that prevent the machine from powering on by running this command:
    vim-cmd vmsvc/get.tasklist VMID
Example output:
(ManagedObjectReference) [
'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-1234567',
'vim.Task:haTask-2-vim.VirtualMachine.consolidateDisks-12345678'
]
  1. Run the vim-cmd vimsvc/task_info task_id command to see the task in more detail.
For example using the data from step 5:
vim-cmd vimsvc/task_info haTask-2-vim.VirtualMachine.createSnapshot-1234567

vim.TaskInfo) {
dynamicType = <unset>,
key = "haTask-2-vim.VirtualMachine.createSnapshot-1234567",
task = 'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-1234567',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.createSnapshot",
descriptionId = "VirtualMachine.createSnapshot",
entity = 'vim.VirtualMachine:2',
entityName = "VMware vCenter Server Appliance5-1",
state = "running",
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 50,
reason = (vim.TaskReasonUser) {
dynamicType = <unset>,
userName = "********",
},
queueTime = "2014-11-09T17:49:01.171933Z",
startTime = "2014-11-09T17:49:01.175521Z",
completeTime = <unset>,
eventChainId = 1234567,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
}
  1. The task is hung and needs to be cancelled, use the command:
    vim-cmd vimsvc/task_cancel task_id
Note: Please note that not all tasks can be canceled. For instance, in the example above we see the snapshot task has the cancelable flag set to false (cancelable = false).
 
  1. Shutdown the virtual machine using the VMID found in Step 3 and run this command:
    vim-cmd vmsvc/power.shutdown VMID
Note: If the virtual machine fails to shut down, run this command to hard power-off the VM:
vim-cmd vmsvc/power.off VMID
Example output:


 

Using esxcli

  1. Log in as root to the ESXi using SSH.
  2. Get a list of running virtual machines, identified by World ID and Display Name by running this command:
    esxcli vm process list

Example output:

 
  1. Take a note of the world ID of the non-responsive virtual machine. For example in step 3 the World ID is 2750434.
  2. Power off the virtual machine from the list by running one of these commands:
    esxcli vm process kill -t=soft -w=WorldID
      or
    esxcli vm process kill -t=hard -w=WorldID
      or
    esxcli vm process kill -t=force -w=WorldID
Note: These command provide different levels of force to stop the virtual machine:
  • Soft is the most graceful
  • Hard performs an immediate shutdown
  • Force should be used as a last resort
  1. If the VM fails to stop even after running the "esxcli vm process kill" command, evacuate all other VMs off the ESXi host and restart the ESXi itself.

 

Using esxtop

Killing a virtual machine using esxtop is equivalent to a forced hard shutdown with esxcli.

  1. Log in as root to the ESXi host using SSH
  2. Show a list of the currently running virtual machines with esxtop
    esxtop
  3. Press "Shift-V" to show only running virtual machines.
  4. Press "F" to add new fields, and then "C" to select the LWID (Leader World Id) field. Press "Enter" to apply the change
  5. To kill one of the virtual machine processes, Press "K" and copy the LWID from the unresponsive VM you wish to kill.

Additional Information