Unable to Power off a Virtual Machine in an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on powering off virtual machines in an ESXi environment.

Unable to power off the virtual machine in an ESXi host.
A virtual machine is not responsive and cannot be stopped or killed/terminated.

Note: If attempting to troubleshoot an unresponsive virtual machine, it is usually preferred to collect performance metrics and then either suspend or crash the virtual machine to collect additional troubleshooting information.

For more information, see:

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Resolution

Follow one of the options mentioned below:

Option 1: Powering off the virtual machine using the vSphere Client

1. Log in to the vCenter Server vSphere Client.
2. Right-click the virtual machine and click Power off. If the power operation fails with "Another task is already in progress", wait for the task to complete or see vCenter operation times out with the error: Operation failed since another task is in progress.
3. If the virtual machine remains running, attempt the same operation from the ESXi host directly.
  - Log into the Host Client directly to the ESXi host running the virtual machine.
  - Right-click on the virtual machine and select Power off.
4. If the virtual machine cannot be powered off gracefully using the client, use one of the command-line options.

Note: From vSphere 6.5, enable Virtual Machine Encryption to protect virtual machine and its files. If encryption is enabled on the virtual machine, ensure the logged in user has Cryptographic Operations privileges. For more information on the Privileges, see Prerequisites and Required Privileges for Encryption Tasks section in the ESXi and vCenter Server 7.0 Documentation.

Option 2: Powering Off a Virtual Machine using ESXCLI:

1. Get a list of running virtual machines, identified by World ID, UUID, Display Name, and path to the .vmx configuration file:
  
  esxcli vm process list
2. Power off the virtual machine from the list by running one of these commands:
  
  esxcli vm process kill -t soft -w WorldID
  or
  esxcli vm process kill -t hard -w WorldID
  or
  esxcli vm process kill -t force -w WorldID
3. Repeat Step 2 and validate that the virtual machine is no longer running.

Option 3: Using the ESXi command-line utility vim-cmd to power off the virtual machine

1. On the ESXi console, enter Shell and log in as root. Refer: Using ESXi Shell in ESXi
2. Get a list of all registered virtual machines, identified by their VMID, Display Name, and path to the .vmx configuration file:
  
  vim-cmd vmsvc/getallvms
3. Get the current state of a virtual machine:
  
  vim-cmd vmsvc/power.getstate VMID
4. Shutdown the virtual machine using the VMID found in Step 2:
  
  vim-cmd vmsvc/power.shutdown VMID
  
  Note: If the virtual machine fails to shut down:
  
  vim-cmd vmsvc/power.off VMID

Option 4: Using esxtop to kill a virtual machine

1. On the ESXi console, enter Tech Support mode and log in as root.
2. Run the esxtop utility by running this command:
  
  esxtop
3. Press c to switch to the CPU resource utilization screen.
4. Press Shift+v to limit the view to virtual machines. This may make it easier to find the Leader World ID in step 7.
5. Press f to display the list of fields.
6. Press c to add the column for the Leader World ID.
7. Identify the target virtual machine by its Name and Leader World ID (LWID).
8. Press k.
9. At the World to kill prompt, type in the Leader World ID from step 7, and press Enter.
10. Wait 30 seconds and validate that the process is no longer listed.

Option 5: Using the kill command to terminate a running virtual machine process. (Warning: This procedure is potentially hazardous to the ESXi host. If improper identification of the appropriate process ID (PID) and the wrong process is killed, it may have unexpected results. )

- On the ESXi console, enter Tech Support mode and log in as root.
- Determine if the virtual machine process is running on the ESXi host by running this command:
  
  ps | grep vmx
  
  The output appears similar to:
  
  7662 7662 vmx /bin/vmx
  7667 7662 vmx /bin/vmx
  7668 7662 mks:VirtualMachineName /bin/vmx
  7669 7662 vcpu-0:VirtualMachineName /bin/vmx
  
  Several rows are returned, one for each vmx process. Identify the parent vmx process for the target virtual machine. The first column contains the PID, and the second contains the parent's PID. Ensure only the parent process is terminated. The parent Process ID (PID) for each process is listed in the second column, identified in this example in bold. Take note of this number for use in the following steps.
  
  Caution: Ensure the line is specific only to the specified affected virtual machine. Continuing this process for a virtual machine other than the one in question may cause downtime for the other virtual machine.
- If the vmx process is listed, terminate the process by running this command:
  
  kill ProcessID
  
  Wait 30 seconds and repeat step 2 to check for the process again.
  If it is not terminated, run this command:
  
  kill -9 ProcessID
  
  Wait 30 seconds and check for the process again.

Note: If the above procedures do not resolve the issue and the ESXi host is responsive, a reboot of the host is required to power off the Virtual Machine.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Powering off an unresponsive virtual machine on an ESXi host
Troubleshooting virtual machine power on task hangs
Failed to power on virtual machine

If a virtual machine cannot be powered off using any of above methods, it usually indicates a problem with the underlying infrastructure, such as the ESXi host or its backing hardware.

If a problem is suspected with the ESXi host that is preventing the shutdown of virtual machines, vMotion all unaffected virtual machines off the host, and force the host to halt with a purple diagnostic screen. For more information, see Using hardware NMI facilities to troubleshoot unresponsive hosts (344096).