Generate .vmem and .vmss files for VMs running on the vSAN cluster

Products

• VMware vSphere ESXi

Issue/Introduction

The steps in this article will capture a memory dump from a VMware virtual machine which is running on a vSAN datastore. This is useful in cases where the Guest OS vendor requires the memory dump for core dump analysis.

Environment

• VMware vSAN 6.x
• VMware vSAN 7.x
• VMware vSAN 8.x

Cause

vSAN datastore is an object store file system, the OSFS namespace will only contain the pointer file of about 118 bytes which points to the associated object within vSAN datastore similar to all the VMDK files and vswap file.

Resolution

Pre-requisites -
+ VM's hardware version should be higher than 10. To check the VM's hardware version > go to host/cluster view > click on VMs option.

lrs;0;[0]
Note: If the hardware version is 10 or lower, when we suspend the VM, we will only get .vmss file. Make sure you backup the VMs before upgrading the hardware version.

To upgrade the hardware version of the VM, first power off the VM > right click VM > Compatibility > Upgrade VM Compatibility.

VM’s hardware version will upgrade to the latest hardware version supported by the environment. 
Refer - ESXi hosts and compatible virtual machine hardware versions list

The following steps will help you to download the vmem file from the vsan datastore which can be shared with the Guest OS vendors.

Option 1 :

If we have a local datastore with free space greater than the memory size of the virtual machine, then

• Browse the datastore, select the .vmem file.
• Next drag and drop the .vmem file from the vSAN datastore to the local datastore.This will initiate a full copy of the pointer file and the associated vSAN object to the local datastore
• Next download the file from the local datastore by either from webclient GUI or WINSCP, this file can be later shared with respective OS vendor for debugging.

 

Option 2 :

If the vSAN cluster does not have a local datastore configured. We will need to follow below steps to download the file.

• Suspend the VM now by – Right click VM > Power > Suspend.
  Refer - Suspending a virtual machine on ESX/ESXi to collect diagnostic information (broadcom.com)
  
  
  
  

  Now, if you navigate to the vSAN directory where the VM is storing its file, you should be able to see both .vmem and .vmss file.
  

  Use cat with the .vmem file to retrieve the object ID.
  

         Then run the command :- /usr/lib/vmware/osfs/bin/objtool open -u ae973167-982f-7dfb-a335-00505603269d

         

 

• Then use “winscp”, logon to the Esxi host, browse to the file path which is /vmfs/devices/vsan/ae973167-982f-7dfb-a335-00505603269d to move the file to the local location or remote location of the machine from which you have logged in .
  

  
  
  

•  Once the file has been copied over to your local system , you will need to add the file extension of .vmem to the file name so that Microsoft or any vendor can use this file for Debugging purpose .

Additional Information

The vmem file generated will have about the same size as the RAM allocated to the VM.