Suspending a virtual machine on ESX/ESXi to collect diagnostic information

Products

VMware vSphere ESXi

Issue/Introduction

When troubleshooting issues related to virtual machine availability or responsiveness on VMware ESXi, it may be necessary to suspend a virtual machine and collect diagnostic information.

For vSphere 6.5 onward, support bundles can include encrypted information from an ESXi host. For more information on what information is included in the support bundles, see Data collected when gathering diagnostic information from vSphere products (2147388)

This article provides several methods for suspending a virtual machine on several versions of VMware ESXi. Deciding whether to crash or suspend a virtual machine is outside the scope of this article.

For more information on:

Troubleshooting, see Troubleshooting a virtual machine that has stopped responding (1007819)

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0

Resolution

Several methods are available to suspend a virtual machine, some of which are applicable to specific versions of VMware ESXi. Choose the method most appropriate to your environment:

ESXi 6.0.x - 7.0.x and vSphere Client
vm-support command-line (ESXi 6.x and 7.0.x)

ESXi 6.0 - 7.0.x VMs on standard SAN, via vSphere Client

Prerequisites

vSphere Client connected to the host or vCenter Server.
The hostd management service must be running.
The virtual machine must be registered in the inventory.

The vSphere Client can be used to suspend a virtual machine, and its datastore browser used to download the suspend state file.

Use the vSphere Client to connect to an ESXi host or vCenter Server as an administrative user.
Select a virtual machine in the inventory.
Open the console tab, and take a screenshot.
Right-click the virtual machine, and select Power - Suspend.
Wait for the Suspend virtual machine task to complete.
Browse the datastore where the virtual machine configuration files are stored.
collect the suspend state files
1. If the virtual machine is hardware version 10 or lower:
  - Locate the .vmss file in the directory of the virtual machine.
  - Right-click the .vmss file, and download it to the client.
2. if the virtual machine is hardware version 11 or newer, or if the virtual machine is running on a virtual volume (VVOL):
  - Locate the .vmss file and the .vmem file in the directory of the virtual machine.
  - Right-click each of the 2 files, and download them to the client.

Note: For VMware vSphere 8.0 refer to VMware Document

ESXi 6.0 - 7.0.x VMs on standard SAN, via vm-support command-line

Prerequisites

Console session open to the ESXi host, either at the local console or via SSH.
The virtual machine must be registered in the inventory.

The vm-support command-line log collection utility supports the use of manifests to create or collect specific information from a host or virtual machine.

Log in to the ESXi Shell as root.

Obtain a list of all registered and running virtual machines using this command:

# vim-cmd vmsvc/getallvms
Identify the affected VM and note down its ID
Suspend the Virtual Machine through CLI using the ID from step 3

# vim-cmd vmsvc/power.suspend VMID

Note: If the incorrect VMID is selected in the above command, it may lead to a production outage. Please ensure to select the correct VMID to suspend the correct VM.

Collect suspend state from a specific virtual machine, plus performance metrics and logs from the host, using this command:

# vm-support --performance --manifests="HungVM:Suspend_VM" --groups="Hardware Logs Network Storage System Userworld VirtualMachines" --vm="</vmfs/volumes/path/to/virtualmachine.vmx>"
A vm-support log bundle is created locally on the host, its filename and location are displayed when complete. Copy this file off the host manually.

ESXi 6.0 - 7.0.x VMs on vSAN, via vSphere Client

Browse the datastore, select the .vmem file.
Drag and drop the .vmem file from the vSAN datastore to a local datastore with sufficient free space.
Download the file from the local datastore using the WebClient GUI or WinSCP.

ESXi 6.0 - 8.0.x VMs on vSAN, via Command-Line

Suspend the VM following the steps mentioned in this. KB.
Log on to the ESXi host using Putty, where the VM in question is registered and running.
Navigate to the VM directory location, e.g.

# cd /vmfs/volumes/vsan:524fcc9a876dcefd-be39dfd52bd37e0e/<VM directory>
Run the command "cat" against the .vmem file:

# cat <file name>.vmem
Retrieve the Object ID information from the output.
Run the command:

# /usr/lib/vmware/osfs/bin/objtool open -u <object ID>
Use WinSCP to log on to the ESXi host and browse to the file path "/vmfs/devices/vsan/<object ID>" to move the file to the local or remote location.
Rename the file extension to .vmem for debugging purposes.

Additional Information

If the virtual machine is a Full Clone desktop in Horizon, it must be placed on Maintenance Mode before proceeding with the preceding steps. Otherwise, if Horizon has the desktops set to Ensure desktops are always powered on, the virtual machine gets powered on as the virtual machine is suspended.

In the VMware Horizon View Administrator, Navigate to Inventory > Desktops.
Right-click the virtual machine and click Enter Maintenance Mode.
Suspend the virtual machine.

If the virtual machine is an Instant Clone desktop in Horizon, you won't be able suspend the virtual machine in vSphere as Horizon will recycle the VM before you can complete your data collection. Instead follow these steps:

Take a snapshot of the VM in vSphere. Select the option to Snapshot the virtual machine's memory.
Follow the steps in Converting a snapshot file to memory dump using the vmss2core tool to convert the vmss to a dmp file.

“vm-support” command in ESX/ESXi to collect diagnostic information
Collecting diagnostic information for VMware ESX/ESXi