Unable to Power off a Virtual Machine in an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on stopping virtual machines in an ESXi environment.

Unable to power off the virtual machine in an ESXi host.
A virtual machine is not responsive and cannot be stopped or killed/terminated.

Note: If attempting to troubleshoot the cause of an unresponsive virtual machine, it is usually preferable to collect performance metrics and then either suspend or crash the virtual machine to collect additional troubleshooting information.

For more information, see:

Resolution

Warning: Follow the sections and steps in this article in order. Do not skip a section or step, as each step may have an impact on the virtual machine.

Powering off the virtual machine using the vSphere Client

Attempt to power off the virtual machine gracefully using the client:

Access and log into the vCenter Server vSphere Client
Right-click the virtual machine and click Power off. If the power operation fails with "Another task is already in progress", wait for the task to complete or see Powering off a virtual machine fails with the error: Cannot power Off: Another task is already in progress or vCenter operation times out with the error: Operation failed since another task is in progress.
If the virtual machine remains running, attempt the same operation from the ESXi host directly.
1. Log into the Host Client directly to the ESXi host running the virtual machine.
2. Right-click on the virtual machine and select Power off.
If the virtual machine cannot be powered off gracefully using the client, use one of the command-line options.

Note: From vSphere 6.5, enable Virtual Machine Encryption to protect virtual machine and its files. If encryption is enabled on the virtual machine, ensure the logged in user has Cryptographic Operations privileges. For more information on the Privileges, see Prerequisites and Required Privileges for Encryption Tasks section in the ESXi and vCenter Server 7.0 Documentation.

Using the ESXi esxcli command to power off a virtual machine

The esxcli command can be used locally or remotely to power off a virtual machine running on ESXi 5.x or later.

Get a list of running virtual machines, identified by World ID, UUID, Display Name, and path to the .vmx configuration file:

esxcli vm process list
Power off the virtual machine from the list:

esxcli vm process kill --type= [soft,hard,force] --world-id= WorldNumber

Note: Three power-off methods are available. Soft is the most graceful, hard performs an immediate shutdown, and force should be used as a last resort.
Repeat Step 2 and validate that the virtual machine is no longer running.

Using esxtop to kill a virtual machine

Use the k command in esxtop to send a signal to and kill a running virtual machine process.

On the ESXi console, enter Tech Support mode and log in as root.
Run the esxtop utility by running this command:

esxtop
Press c to switch to the CPU resource utilization screen.
Press Shift+v to limit the view to virtual machines. This may make it easier to find the Leader World ID in step 7.
Press f to display the list of fields.
Press c to add the column for the Leader World ID.
Identify the target virtual machine by its Name and Leader World ID (LWID).
Press k.
At the World to kill prompt, type in the Leader World ID from step 7, and press Enter.
Wait 30 seconds and validate that the process is no longer listed.

Using the ESXi command-line utility vim-cmd to power off the virtual machine

On the ESXi console, enter Tech Support mode and log in as root.
Get a list of all registered virtual machines, identified by their VMID, Display Name, and path to the .vmx configuration file:

vim-cmd vmsvc/getallvms
Get the current state of a virtual machine:

vim-cmd vmsvc/power.getstate VMID
Shutdown the virtual machine using the VMID found in Step 2:

vim-cmd vmsvc/power.shutdown VMID

Note: If the virtual machine fails to shut down:

vim-cmd vmsvc/power.off VMID

Note: If the above procedures did not resolve the issue and the ESXi host is responsive, it may be required to reboot the host to put the virtual machine in a powered-off state. Migrating all the virtual machines to other ESXi with vMotion to avoid downtime may also be done.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Powering off an unresponsive virtual machine on an ESXi host
Troubleshooting virtual machine power on task hangs
Failed to power on virtual machine

Sending signals on ESXi to power off the virtual machine

A virtual machine can be halted from the command line by sending signals to the process.

Warning: This procedure is potentially hazardous to the ESXi host. If improper identification of the appropriate process ID (PID) and the wrong process is killed, it may have unexpected results.

Use the kill command to send a signal to, and terminate, a running virtual machine process.

On the ESXi console, enter Tech Support mode and log in as root.
Determine if the virtual machine process is running on the ESXi host by running this command:

ps | grep vmx

The output appears similar to:

7662 7662 vmx /bin/vmx
7667 7662 vmx /bin/vmx
7668 7662 mks:VirtualMachineName /bin/vmx
7669 7662 vcpu-0:VirtualMachineName /bin/vmx

Several rows are returned, one for each vmx process. Identify the parent vmx process for the target virtual machine. The first column contains the PID, and the second contains the parent's PID. Ensure only the parent process is terminated. The parent Process ID (PID) for each process is listed in the second column, identified in this example in bold. Take note of this number for use in the following steps.

Caution: Ensure the line is specific only to the specified affected virtual machine. Continuing this process for a virtual machine other than the one in question may cause downtime for the other virtual machine.
If the vmx process is listed, terminate the process by running this command:

kill ProcessID
Wait 30 seconds and repeat step 2 to check for the process again.
If it is not terminated, run this command:

kill -9 ProcessID
Wait 30 seconds and check for the process again.

If a virtual machine cannot be powered off using any of these methods, it usually indicates a problem with the underlying infrastructure, such as the ESXi host or its backing hardware.

If a problem is suspected with the ESXi host that is preventing the shutdown of virtual machines, vMotion all unaffected virtual machines off the host, and force the host to halt with a purple diagnostic screen. For more information, see Using hardware NMI facilities to troubleshoot unresponsive hosts (344096).

For more information, see: