Virtual machines become unresponsive when generating a log bundle in vSphere 5.0 and 5.1 with the error: Unsync core dump requested; not a real fault

Products

• VMware

Issue/Introduction

When you generate a log bundle in vSphere 5.0 or 5.1 and select the HungVM manifest or the Select All manifests option, virtual machines on the ESXi/ESX host or cluster will become unresponsive. This article provides information on the steps that can lead to this issue and also the steps required to avoid it.

Symptoms:

• When you generate a log bundle in a vSphere 5.0 or 5.1 environment, virtual machines may become unresponsive.
• The hostd.log file (located at /var/log) contains entries similar to these, indicating a call to the vm-support log collection script:
  
  
  12:14:39.795Z [2A181B90 info 'Snmpsvc'] UpdateStats: report cimom converter stats completed
  12:14:40.731Z [29D81B90 info 'TaskManager' opID=27c2fe17-4a] Task Created : haTask--vim.SessionManager.acquireGenericServiceTicket-345364944
  12:14:40.731Z [2A5FEB90 verbose 'HaCgiServiceManagerImpl' opID=27c2fe17-4a] Ticket created: ((null) https://esx01.local/cgi-bin/vm-support.cgi?...
  12:14:40.731Z [2A5FEB90 info 'TaskManager' opID=27c2fe17-4a] Task Completed : haTask--vim.SessionManager.acquireGenericServiceTicket-345364944 Status success
  12:14:40.747Z [2A57DB90 verbose 'HttpSvc.HTTPService.HttpConnection'] User agent is 'Jakarta Commons-HttpClient/3.1'
  12:14:40.748Z [2A57DB90 verbose 'HaCgiServiceManagerImpl'] Ticket valid: ***86333: (GET http://localhost/cgi-bin/vm-support.cgi?manifests=System:Base System:EsxImage System:GPUResource System:Modules System:ResourceGroups System:VMKernel System:VMKernel_Cores CrashDumps:userworld Hardware:base Hardware:usb Configuration:System Installer:base ActiveDirectory:base Userworld:FDM Userworld:HostAgent Userworld:ProcessInformation hostProfiles:SystemImageCacheHostProfile hostProfiles:hostProfiles VirtualMachines:VirtualMachineStats VirtualMachines:base VirtualMachines:corefiles VirtualMachines:diskinfo VirtualMachines:logs FileSystem:base FaultTolerance:Base IntegrityChecks:md5sums Storage:FCoE Storage:Multipathing Storage:NAS Storage:base Storage:iodm Storage:iscsi Logs:System Network:base Network:dvs Network:tcpip Network:vxlan CIM:base System:VMKernel_Old_Cores PerformanceSnapshot:vsi VirtualMachines:SnapshotsAndMemoryImages FileSystem:vmfsheader Testing:test-esx-logs HungVM
  12:14:40.749Z [2A57DB90 info 'SysCommandPosix'] ForkExec(/usr/bin/sh) 4761018
  12:14:40.749Z [2A57DB90 verbose 'CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command started.
  .........
  00:43:45.586Z [52A40B90 verbose 'CgiResponseWriter(/cgi-bin/vm-support.cgi)'] Close
  00:43:45.586Z [52A40B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Complete (processed 3532226303 bytes)
  00:43:45.587Z [52A40B90 verbose 'CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Sent Ok response
  00:43:45.587Z [52A40B90 verbose 'CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command finished. <<<<<<<<<<<Completion of the command<br>00:43:51.878Z [FFDE8AC0 verbose 'Proxysvc Req55880'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:60072)
  .........
  YYYY-10-22T01:17:21.325Z [22780B90 verbose 'HaCgiServiceManagerImpl' opID=71E9433E-00000DF8-c] Ticket created: ((null) https://<hostname>/cgi-bin/vm-support.cgi?manifests=HungVM:Coredump_VM HungVM:Send_NMI_To_Guest HungVM:Suspend_VM&vm=/vmfs/volumes/50762f6f-29707a12-7ce3-bc305bee84f5/<vmwarename>/<vmwarename.vmx) - ***df7ff</font>
  2013-10-22T01:17:21.360Z [22780B90 verbose 'HaCgiServiceManagerImpl'] Ticket valid: ***df7ff: (GET http://localhost/cgi-bin/vm-support.cgi?manifests=HungVM:Coredump_VM HungVM:Send_NMI_To_Guest HungVM:Suspend_VM&vm=/vmfs/volumes/50762f6f-29707a12-7ce3-bc305bee84f5/vmname/vmname1.vmx) == ((null)
  

• The vmware.log file (located in the virtual machine home directory) contains entries similar to:
  
  22:28:54.896Z| vmx| I120: MonitorVMMCoreRequest: ***
  22:28:54.896Z| vmx| I120: MonitorVMMCoreRequest: Unsync core dump requested; not a real fault
  22:28:54.896Z| vmx| I120: MonitorVMMCoreRequest: ***
  
  
• The vmdumper.log file (located in the virtual machine home directory) contains entries that list the actions are taken while the virtual machine is failing:
  
  Running /sbin/vmdumper 1038745 screenshot
  Grabbing screenshot...
  Running /sbin/vmdumper 1038745 samples_on
  Turning VM samples on...
  Running /sbin/vmdumper 1038745 samples_off
  Turning VM samples off...
  Running /sbin/vmdumper 1038745 unsync
  Dumping unsync cores...
  Running /sbin/vmdumper 1038745 sync
  Dumping sync cores...
  Running /sbin/vmdumper 1038745 vmx
  Dumping vmx core...
  Running /sbin/vmdumper 1038745 vmx_force
  Dumping vmx core (force)...
  Running /sbin/vmdumper 1038745 screenshot
  Grabbing screenshot...
  Running /sbin/vmdumper 1038745 samples_on
  Turning VM samples on...
  Running /sbin/vmdumper 1038745 samples_off
  Turning VM samples off...
  Running /sbin/vmdumper 1038745 nmi
  Sending NMI to guest...
  Running /sbin/vmdumper 1038745 screenshot
  Grabbing screenshot...
  
  

Cause

This issue occurs when you perform these steps to collect logs:

Warning: Selecting HungVM option during log collection will by design crash the virtual machine. VMware Technical Support may request that you create a virtual machine core dump to troubleshoot an issue in cases where logs do not provide enough information.When these steps are complete, all running virtual machines on the selected ESXi/ESX hosts become unresponsive and create a core dump.

From the vSphere Web Client:

1. Launch the vSphere Web Client and connect to vCenter Server.
2. Under Inventory Lists, click vCenter Servers.
3. Click the vCenter Server that contains the ESXi/ESX hosts from which you want to collect logs.
4. Click the Monitor tab and click System Logs.
5. Click Export System logs.
6. Select the ESXi/ESX server(s) you want to collect logs from and click Next.
   
   Note: You see various manifest options, some of which are pre-selected.
   
   
7. Click Select All.
   
   Note: By default HungVM is deselected. Clicking Select All enables the HungVM manifest. This causes virtual machines to become unresponsive.
   
   
8. Click Generate log bundle.
9. Select Download log bundle and then select Finish.

From the vSphere C# Client:

1. Launch the vSphere C# Client: and connect to vCenter Server.
2. Highlight a virtual machine
3. Click File, Export and Export System Logs.
4. Select the HungVM checkbox.
5. Click Next and then Finish.

For additional options to select HungVM option during Log Collection, see Crashing a virtual machine on ESX/ESXi to collect diagnostic information (2005715).

Resolution

The HungVM option is available in vSphere 5.0 and 5.1 as described in the Cause section.

To avoid crashing virtual machines during log collection ensure that the HungVM option is deselected when you generate a log bundle in vSphere 5.0 or 5.1.


For more information, see Collecting diagnostic information for VMware ESX/ESXi (653).

Additional Information

Crashing a virtual machine on ESX/ESXi to collect diagnostic information
vSphere 5.x でログ バンドルを生成しているとき、次のエラーによって仮想マシンが応答しなくなる。 Unsync core dump requested; not a real fault
在 vSphere 5.0 和 5.1 中生成日志包时虚拟机无响应并显示以下错误： 请求的核心转储未同步；不是实际故障
Collecting diagnostic information for VMware ESX/ESXi