Unable to Power off a Virtual Machine in an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on stopping virtual machines in an ESXi environment.

Note: If you are attempting to troubleshoot the cause of an unresponsive virtual machine, it is usually preferable to collect performance metrics and then either suspend or crash the virtual machine to collect additional troubleshooting information.

For more information, see:

Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926).
Troubleshooting a virtual machine that has stopped responding (1007819).

Symptoms:

Unable to power off the virtual machine in an ESXi host.
A virtual machine is not responsive and cannot be stopped or killed.

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 8.0.x

Resolution

Solution when Unable to power off a virtual machine in an ESXi host:

Warning: Follow the sections and steps in this article in order. Do not skip a section or step, as each step may have an impact on the virtual machine.

Powering off the virtual machine using the vSphere Client/vSphere Web Client
Using the ESXi esxcli command to power off a virtual machine
Using esxtop to kill a virtual machine
Using the ESXi command-line utility vim-cmd to power off the virtual machine

Powering off the virtual machine using the vSphere Client/vSphere Web Client

Attempt to power off the virtual machine gracefully using the client:

Connect the vSphere Client or vSphere Web Client to VMware vCenter Server.
Right-click the virtual machine and click Power off. If the power operation fails with Another task is already in progress, wait for the task to complete or see Powering off a virtual machine fails with the error: Cannot power Off: Another task is already in progress (1027040) or vCenter operation times out with the error: Operation failed since another task is in progress (1004790).
If the virtual machine remains running, attempt the same operation from the ESXi host directly.
Connect the vSphere Client or VMware Host Client directly to the ESXi host running the virtual machine.
Right-click on the virtual machine and select Power off.

Note: From vSphere 6.5, you can enable Virtual Machine Encryption to protect virtual machine and its files.

If encryption is enabled on the virtual machine then you must have a set of Cryptographic Operations privileges failing which you will not be able to power off the virtual machine.

For more information on the Privileges, see Prerequisites and Required Privileges for Encryption Tasks section in the ESXi and vCenter Server 6.5 Documentation.

If the virtual machine cannot be powered off gracefully using the client, use one of the command-line options.

Using the ESXi esxcli command to power off a virtual machine

The esxcli command can be used locally or remotely to power off a virtual machine running on ESXi 5.x or later.

Get a list of running virtual machines, identified by World ID, UUID, Display Name, and path to the .vmx configuration file by running this command:

esxcli vm process list
Power off the virtual machine from the list by running this command:

esxcli vm process kill --type= [soft,hard,force] --world-id= WorldNumber

Notes:
- Three power-off methods are available. Soft is the most graceful, hard performs an immediate shutdown, and force should be used as a last resort.
Repeat Step 2 and validate that the virtual machine is no longer running.

Using esxtop to kill a virtual machine

You can use the k command in esxtop to send a signal to, and kill, a running virtual machine process.

On the ESXi console, enter Tech Support mode and log in as root.
Run the esxtop utility by running this command:

esxtop
Press c to switch to the CPU resource utilization screen.
Press Shift+v to limit the view to virtual machines. This may make it easier to find the Leader World ID in step 7.
Press f to display the list of fields.
Press c to add the column for the Leader World ID.
Identify the target virtual machine by its Name and Leader World ID (LWID).
Press k.
At the World to kill prompt, type in the Leader World ID from step 7, and press Enter.
Wait 30 seconds and validate that the process is no longer listed.

Using the ESXi command-line utility vim-cmd to power off the virtual machine

On the ESXi console, enter Tech Support mode and log in as root.
Get a list of all registered virtual machines, identified by their VMID, Display Name, and path to the .vmx configuration file by running this command:

vim-cmd vmsvc/getallvms
Get the current state of a virtual machine by running this command:

vim-cmd vmsvc/power.getstate VMID
Shutdown the virtual machine using the VMID found in Step 2 and run this command:

vim-cmd vmsvc/power.shutdown VMID

Note: If the virtual machine fails to shut down, run this command:

vim-cmd vmsvc/power.off VMID

Note: If the above procedures did not resolve the issue and the ESXi host is responsive, it may need to be rebooted to put the virtual machine in a powered-off state. You could migrate all the virtual machines to other ESXi with vMotion to avoid downtime.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Powering off an unresponsive virtual machine on an ESXi host
Troubleshooting virtual machine power on task hangs
Failed to power on virtual machine
Sending signals on ESXi to power off the virtual machine

A virtual machine can be halted from the command line by sending signals to the process.

Warning: This procedure is potentially hazardous to the ESXi host. If you do not identify the appropriate process ID (PID) and kill the wrong process, it may have unexpected results. If you are not comfortable with the following procedure, file a support request with Broadcom Support and note this Knowledge Base article ID (320877) in the problem description. For more information, please visit https://support.broadcom.com

You can use the kill command to send a signal to, and terminate, a running virtual machine process.

On the ESXi console, enter Tech Support mode and log in as root.
Determine if the virtual machine process is running on the ESXi host by running this command:

ps | grep vmx

The output appears similar to:

7662 7662 vmx /bin/vmx 7667 7662 vmx /bin/vmx 7668 7662 mks:VirtualMachineName /bin/vmx 7669 7662 vcpu-0:VirtualMachineName /bin/vmx

Several rows are returned, one for each vmx process. Identify the parent vmx process for the target virtual machine. The first column contains the PID, and the second contains the parent's PID. Ensure you terminate only the parent process. The parent Process ID (PID) for each process is listed in the second column, identified in this example in bold. Take note of this number for use in the following steps.

Caution: Ensure that you identify the line specific only to the virtual machine you are attempting to repair. If you continue this process for a virtual machine other than the one in question, you can cause downtime for the other virtual machine.
If the vmx process is listed, terminate the process by running this command:

kill ProcessID
Wait 30 seconds and repeat step 2 to check for the process again.
If it is not terminated, run this command:

kill -9 ProcessID
Wait 30 seconds and check for the process again.

If a virtual machine cannot be powered off using any of these methods, it usually indicates a problem with the underlying infrastructure, such as the ESXi host or its backing hardware.

If a problem is suspected with the ESXi host that is preventing the shutdown of virtual machines, vMotion all unaffected virtual machines off the host, and force the host to halt with a purple diagnostic screen. For more information, see Using hardware NMI facilities to troubleshoot unresponsive hosts (1014767).

For more information, see: