Virtual Machine enters into "Suspended" state while using "export system logs" option from vCenter

Products

• VMware vCenter Server

Issue/Introduction

• A Virtual Machine (VM) unexpectedly transitions into a Suspended state.

• Exporting virtual machines logs from vCenter Server using the Export Logs function and selecting the Select All option may result in virtual machine state change.

  • This selection includes options like VirtualMachines:CoreDumpHung and HungVM, which include embedded commands such as Suspend_VM and Send_NMI_To_Guest.

• The targeted virtual machines immediately transition into a suspended state while the log bundle downloads.

• An event is recorded on the ESXi host where the VM is running stating "VM-Name is suspended" but the system does not explicitly indicate that the suspension was caused by the log export task.

• The following log entries are observed within the vmware.log file, found in the virtual machine's directory on the ESXi Host where it is registered:

  • /vmfs/volumes/<Datastore Name>/<VM Name>/vmware.log:

    • YYYY-MMDDThh:mm:ss In(05) vmx - SUSPEND: Start suspend (flags=0)
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=##################
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - VMMon_VSCSIStopVports: No such target on adapter
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Closing all the disks of the VM.
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Closing disk 'scsi0:0'
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - DISKLIB-CBT   : Shutting down change tracking for untracked fid 252699447.
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - DISKLIB-CBT   : Successfully disconnected CBT node.
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/<datastore>/<vm_folder>/vmname-flat.vmdk" : closed.
YYYY-MMDDThh:mm:ss No(00) vcpu-0 - CheckpointTiming save: Preparation took 884421 us
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Progress -1% (msg.checkpoint.saveStatus)
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Checkpointed in VMware ESX, 8.0.3, build-24859861, Linux Host
YYYY-MMDDThh:mm:ss No(00) vcpu-0 - ConfigDB: Setting checkpoint.vmState = "vnname.vmss"
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - Vix: [mainDispatch.c:4211]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1875, success=1 additionalError=0
YYYY-MMDDThh:mm:ss In(05) vcpu-0 - SUSPEND: Completed suspend: 'Operation completed successfully' (0)
YYYY-MMDDThh:mm:ss In(05) vmx - MonitorVMMCoreRequest: ********************************************
YYYY-MMDDThh:mm:ss In(05) vmx - MonitorVMMCoreRequest: Sync core dump requested; not a real fault
YYYY-MMDDThh:mm:ss In(05) vmx - MonitorVMMCoreRequest: ********************************************
YYYY-MMDDThh:mm:ss In(05) vmx - MonitorActionSetBit: vcpu-0 id 52
YYYY-MMDDThh:mm:ss In(05) vmx -
YYYY-MMDDThh:mm:ss Cr(01) vmx - PANIC: VERIFY bora/vmcore/vmx/main/monitorAction.c:662 bugNr=10871
YYYY-MMDDThh:mm:ss Wa(03) vmx - A core file is available in "/var/core/vmx-zdump.000"

• The following log entries are observed in /var/run/log/hostd.log in ESXi Host:

  • YYYY-MMDDThh:mm:ss In(166) Hostd[2100171]: [Originator@6876 sub=Vimsvc.TaskManager opID=<Operation_ID> sid=52a3948e user=vpxuser:<Username>] Task Created : haTask--vim.SessionManager.acquireGenericServiceTicket-27605365
YYYY-MMDDThh:mm:ss In(166) Hostd[2100156]: [Originator@6876 sub=Vimsvc.TaskManager opID=<Operation_ID> user=vpxuser:<Username>] Task Completed : haTask--vim.SessionManager.acquireGenericServiceTicket-27605365 Status success
YYYY-MMDDThh:mm:ss In(166) Hostd[2100165]: [Originator@6876 sub=Vimsvc.CgiServiceManager opID=CGI-server-0503 sid=#########] Validating CGI ticket for URL '/cgi-bin/vm-support.cgi?manifests=VirtualMachines:CoreDumpHung HungVM:Send_NMI_To_Guest HungVM:Suspend_VM
YYYY-MMDDThh:mm:ss In(166) Hostd[2100165]: [Originator@6876 sub=Vimsvc.CgiServiceManager opID=CGI-server-0503 sid=#########] Ticket URL pair is valid

• From /var/run/log/envoy-access.log 

  • YYYY-MMDDThh:mm:ss In(166) envoy-access[2099889]: POST /cgi-bin/vm-support.cgi?manifests=VirtualMachines:CoreDumpHung%20HungVM:Send_NMI_To_Guest%20HungVM:Suspend_VM 200 via_upstream - 132 36529303 - 600494 71186 529307 <vCenter_IP>:43654 HTTP/1.1 TLSv1.2 <ESXi_host_IP>:443 127.0.0.1:19986 HTTP/1.1 - 127.0.0.1:8303 - -

Cause

• The Virtual Machine entered into suspended state intentionally by the ESXi host's diagnostic subsystem (vmdumper) to capture a consistent memory state. This occurs when an administrator initiates a log collection like the one listed in Generating a VMkernel zdump manually from a dump file in ESXi host
• Selecting the "Select All" option bundles routines like VirtualMachines:CoreDumpHung and HungVM. These routines contain direct commands Suspend_VM and Send_NMI_To_Guest designed to force a core dump.

 

Resolution

This is expected behavior when the "select all" option is selected, as it includes VirtualMachines:CoreDumpHung and HungVM actions.

1. Resume the VM:

1. • • Log in to the vSphere Client or ESXi Host Client.
     • Right-click the suspended Virtual Machine.
     • Select Power > Power On (or Resume).
       
       
2. Verify Timestamps (Optional):
   • • Compare the "Date Modified" timestamp of the vmdumper.log file in the VM's datastore folder with the suspension time in vCenter events. A match confirms the dump generation caused the suspension.

• Workaround :To collect logs without suspending the VM in the future:

• • • Use the standard Export System Logs workflow.
    • Ensure options related to "Hung VM," "Memory Checkpoint," or "Performance Snapshots" are deselected unless the VM is actually unresponsive and a core dump is required for root cause analysis.