Suspending a virtual machine on ESX/ESXi to collect diagnostic information

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

When troubleshooting issues related to virtual machine availability or responsiveness on VMware ESXi, it may be necessary to suspend a virtual machine and collect diagnostic information.

For vSphere 6.5 onward, support bundles can include encrypted information from an ESXi host. For more information on what information is included in the support bundles, see Data collected when gathering diagnostic information from vSphere products (2147388)

This article provides several methods for suspending a virtual machine on several versions of VMware ESXi. Deciding whether to crash or suspend a virtual machine is outside the scope of this article.

For more information on:

Troubleshooting, see Troubleshooting a virtual machine that has stopped responding (1007819)

Resolution

Several methods are available to suspend a virtual machine, some of which are applicable to specific versions of VMware ESXi. Choose the method most appropriate to your environment:

Suspend and Collect log bundle using "Export System Logs" option via vSphere Client

The vSphere Client can be used to suspend a virtual machine and collect the data using "Export System Logs" option from virtual machine's Action Menu.

Prerequisites

vSphere Client connected to the vCenter Server.
The ESXi Server and the Virtual Machine must be in connected state

Procedure

Use the vSphere Client to connect to an ESXi host or vCenter Server as an administrative user.
Select a virtual machine in the inventory.
Right-click the virtual machine, and select "Export System Logs".
Wait for the workflow to finish and download the Diagnostic Log bundle.

Suspend and collect the data for VMs on non vSAN Datastore using via vm-support command-line

Prerequisites

Console session open to the ESXi host, either at the local console or via SSH.
The virtual machine must be registered in the inventory.

The vm-support command-line log collection utility supports the use of manifests to create or collect specific information from a host or virtual machine.

Log in to the ESXi Shell as root.

Collect suspend state from a specific virtual machine, plus performance metrics and logs from the host, using this command:

# vm-support --performance --manifests="HungVM:Suspend_VM" --groups="Hardware Logs Network Storage System Userworld VirtualMachines" --vm="</vmfs/volumes/path/to/virtualmachine.vmx>"

A vm-support log bundle is created locally on the host, its filename and location are displayed when complete. Copy this file off the host manually.

Suspending VMs using vSphere Client

Prerequisites

vSphere Client connected to the host or vCenter Server.
The hostd management service must be running.
The virtual machine must be registered in the inventory.

Procedure

Use the vSphere Client to connect to an ESXi host or vCenter Server as an administrative user.
Select a virtual machine in the inventory.
Open the console tab, and take a screenshot.
Right-click the virtual machine, and select Power - Suspend.
Wait for the Suspend virtual machine task to complete.

Suspending VMs using command line options

Prerequisites

Console session open to the ESXi host, either at the local console or via SSH.
The virtual machine must be registered in the inventory.

1. Log in to the ESXi Shell as root.
2. Obtain a list of all registered and running virtual machines using this command:
```
# vim-cmd vmsvc/getallvms
```
3. Identify the affected VM and note down its ID
4. Suspend the Virtual Machine through CLI using the ID from step 3
```
# vim-cmd vmsvc/power.suspend VMID
```

Collecting the memory dump manually for VMs via vSphere Client

Prerequisites

vSphere Client connected to the host or vCenter Server.
The hostd management service must be running.
The virtual machine must be registered in the inventory.

Procedure

Use one of the method listed above to Suspend the virtual machine
Browse the datastore where the virtual machine configuration files are stored.
collect the suspend state files
If the virtual machine is hardware version 10 or lower:
1. Locate the .vmss file in the directory of the virtual machine.
2. Right-click the .vmss file, and download it to the client.
if the virtual machine is hardware version 11 or newer, or if the virtual machine is running on a virtual volume (VVOL):
1. Locate the .vmss file and the .vmem file in the directory of the virtual machine.
2. Right-click each of the 2 files, and download them to the client.

Collecting the memory dump manually for VMs on vSAN Datastore via command-line/WinSCP

Suspend the VM following the steps listed above
Log on to the ESXi host using Putty, where the VM in question is registered and running.

Navigate to the VM directory location, e.g.

# cd /vmfs/volumes/vsan:524fcc9a876dcefd-be39dfd52bd37e0e/<VM directory>

Run the command "cat" against the .vmem file:
```
# cat <file name>.vmem
```
Retrieve the Object ID information from the output.

Run the command:

# /usr/lib/vmware/osfs/bin/objtool open -u <object ID>

Use WinSCP to log on to the ESXi host and browse to the file path "/vmfs/devices/vsan/<object ID>" to move the file to the local or remote location.
Rename the file extension to .vmem for debugging purposes.

Additional Information

If the virtual machine is a Full Clone desktop in Horizon, it must be placed on Maintenance Mode before proceeding with the preceding steps. Otherwise, if Horizon has the desktops set to Ensure desktops are always powered on, the virtual machine gets powered on as the virtual machine is suspended.

In the VMware Horizon View Administrator, Navigate to Inventory > Desktops.
Right-click the virtual machine and click Enter Maintenance Mode.
Suspend the virtual machine.

If the virtual machine is an Instant Clone desktop in Horizon, you won't be able suspend the virtual machine in vSphere as Horizon will recycle the VM before you can complete your data collection. Instead follow these steps:

Take a snapshot of the VM in vSphere. Select the option to Snapshot the virtual machine's memory.
Follow the steps in Converting a snapshot file to memory dump using the vmss2core tool to convert the vmss to a dmp file.

“vm-support” command in ESX/ESXi to collect diagnostic information
Collecting diagnostic information for VMware ESX/ESXi