What to collect when a virtual machine is hung or frozen on an ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

This article is meant to provide guidance on which steps to take when experiencing virtual machine being unresponsive, in order to narrow the issue down further and collect the necessary information to allow analyzing what caused the behaviour.

Note: A major part of the required information consists of dynamic data (as opposed to static logs) which will only be available while the virtual machine is still in the unresponsive state, and therefore needs to be collected immediately, before steps are taken to remediate the VM and/or reporting the issue to the Broadcom support team.

Environment

• VMware ESXi 7.0.x
• VMware ESXi 8.0.x
• VMware ESX 9.x

Resolution

1. Confirming the VM state

   1. Start with trying to ping (ICMP) the virtual machine from another VM in the same network and on the same host. If there is a response from the VM it is not frozen, but there could be a network issue outside of the virtual network. To narrow this down further, ping from other system further away (on a different host, or even in a different environment/network).
   2. If the VM does not respond to ICMP even from a VM on the same host and in the same network, open the vSphere Client remote console for the VM and confirm that the VM is not currently in a panic or bluescreen (BSOD).
   3. If the VM is not in a panic/BSOD, verify if it does respond to any inputs. Otherwise involve the guest OS vendor.
   4. If the VM does not take any inputs via vSphere Client remote console either, send a CTRL+ALT+DEL via the console. This will set a marker that can later be identified in the VM memory.
   
   
   

2. Collecting the required data

   1. Identify the world ID (WID) of the virtual machine:

      # vmdumper -l | grep <vm_name> | awk '{print $1}'

      Note it down, it will be needed in the next and later steps.
   2. Enable VMSample logging for the VM using:

      # vmdumper <WID> samples_on

      Wait for 4 to 5 minutes before moving on to the next step, to allow a reasonable number of sample traces being logged to the vmware.log.
   3. Collect the ESXi performance counters by running esxtop in batch mode to confirm if the VM process is using any resources or if it might be stuck due to non-available resources:

      # esxtop -ba -d 2 -n 60 | gzip -9c > /vmfs/volumes/<datastore>/perfstats_$(date +"%Y_%m_%d_%I_%M_%p").csv.gz

      This runs for 2 minutes and collecting 1 set of all performance counters every 2 seconds. Wait until this has finished, then move on to the next step.
   4. Now go in vSphere Client and right-click the virtual machine, then select Power → Suspend to suspend the virtual machine. This will write down the memory state of the VM to in the VM folder in 1 or 2 files, depending on the virtual hardware version of the VM. There will always be a <vm_name>-###.vmss file, but for older virtual hardware versions (below 11) there will also be a <vm_name>-###.vmem file.
      These suspend state files contain the content of the memory used by the virtual machine, including the state of its virtual CPUs, and can later be used with vmss2core to extract a guest memory dump or guest cores, depending on the operating system.
   5. Go back in the command shell of the ESXi host the virtual machine is registered on and change into the VM directory:

      # cd /vmfs/volumes/<datastore>/<vm_folder>/

      The next step depends on if the virtual machine is encrypted or not:
      1. If the VM is not using encryption, create copies of the suspend state files:

         # cp <vm_name>*.vmss <vm_name>.vmss.pck
         # cp <vm_name>*.vmem <vm_name>.vmem.pck

      2. If the VM is encrypted, run the following 2 commands to create decrypted copies of the suspend state files:

         # crypto-util encobj decrypt --related <vm_name> <vm_name>*.vmss <vm_name>.vmss.pck
         # crypto-util encobj decrypt --related <vm_name> <vm_name>*.vmem <vm_name>.vmem.pck

         Note: For more information, refer to Unable to convert a Windows 11 VM snapshot to a memory dump.

   6. Export the MD5 checksums of both files:

      # md5sum <vm_name>.vmss.pck > <vm_name>.vmss.md5sum
      # md5sum <vm_name>.vmem.pck > <vm_name>.vmem.md5sum

      If memory corruption is found while analyzing the VM memory, this will allow confirmation if the corruption was already present or if there was damage to the files during transfer.
   7. Compress the files into an archive:

      # tar czf <vm_name>_suspend.tgz <vm_name>.vmem.pck <vm_name>.vmss.pck <vm_name>.vmss.md5sum <vm_name>.vmem.md5sum

      and download the archive via SCP or using the datastore browser.
   8. Now go back in vSphere Client and unsuspend the virtual machine by right-clicking it again and selecting Power → Power On.
   9. Once the virtual machine is back in running state, move back to the ESXi command shell and query the VMs world ID again. Due to the suspend/resume it has received a new WID, hence run the command from step 1 once more:

      # vmdumper -l | grep <vm_name> | awk '{print $1}'

      Once again, note down the WID.
   10. Using the new WID, run the following commands to collect the VMX cores:

       # vmdumper -l | grep <vm_name> | awk '{print $1}'
       # vmdumper <WID> unsync
       # vmdumper <WID> sync
       # vmdumper <WID> vmx
       # vmdumper <WID> vmx_force

       These allow to isolate if the VMX world (aka the virtual machine process itself) might have been unresponsive and if so, what caused it.
   11. Once the last vmdumper command has finished running, export a log bundle of the ESXi host. See Collecting diagnostic information for VMware ESXi using vSphere Client for more information.
   12. Contact Broadcom support and provide the collected data:
       • The ESXTOP batch trace file created in step 3
       • The archive with the suspend state files from steps 4 to 7
       • The log bundle of the ESXi host created in step 11