Exporting system logs for any virtual machine fails via vCenter UI and downloads an invalid file starting version 8.0u3.

Products

• VMware vCenter Server 8.0

Issue/Introduction

On attempting to download/export the system logs for any virtual machine:

Login to vCenter UI -> Right-Click any Virtual machine > Export System Logs,
it would export an invalid file of few KBs and fail immediately.

The export system logs option is greyed out for the VMs from the ESXi host UI.

Below is the screenshot for the same:

Similarly,  Exporting system logs for a virtual machine with "Send_NMI_To_Guest" option from vSphere Client results in downloading an invalid file that does not contain any data.

 

Below is the log snippet from the /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

[YYYY-MM-DD] [ERROR] nio-127.0.0.1-5090-exec-9007  c.v.vsphere.client.h5.admin.controllers.ExportLogsController
     FileId parameter is empty.
[YYYY-MM-DD] [ERROR] nio-127.0.0.1-5090-exec-9007  c.v.vsphere.client.h5.admin.controllers.ExportLogsController
     Error while downloading logs. java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:1080
/external-vecs/443/cgi-bin/vm-support.cgi?manifests=
        at sun.reflect.GeneratedConstructorAccessor3066.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at com.vmware.vise.util.http.ConnectionManager.getResponseBodyAsStream(ConnectionManager.java:341)
        at com.vmware.vise.util.http.SimpleHttpClient.executeMethodResponseAsStream(SimpleHttpClient.java:223)
        at com.vmware.vise.util.http.SimpleHttpClient.executeMethodResponseAsStream(SimpleHttpClient.java:244)
        at com.vmware.vsphere.client.logbundle.HttpZipEntry.connect(HttpZipEntry.java:211)
        at com.vmware.vsphere.client.logbundle.UrlZipEntry.write(UrlZipEntry.java:165)
        at com.vmware.vsphere.client.h5.admin.controllers.ExportLogsController.writeHostLogsToStream(ExportLogsController.java
:608)
        at com.vmware.vsphere.client.h5.admin.controllers.ExportLogsController.access$500(ExportLogsController.java:82)
        at com.vmware.vsphere.client.h5.admin.controllers.ExportLogsController$1.run(ExportLogsController.java:372)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.vmware.vise.util.concurrent.ExecutorUtil$LatchRunnable.run(ExecutorUtil.java:1301)
        at com.vmware.vise.util.concurrent.ExecutorUtil.executeTasks(ExecutorUtil.java:325

Environment

VMware vCenter Server 8.0.3x

Cause

This is a known issue affecting the 8.0 U3 version of vCenter Servers.

Resolution

Engineering is aware of this issue and this will be resolved in the future release.

Workaround:
Log in to ESXi host through SSH or console as root and follow the below steps to generate the log bundles for specific VMs:

vm-support --listvms

Example output:
/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/New Virtual Machine/New Virtual Machine.vmx (Registered)
/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/Test VMachine/Test VMachine.vmx (Registered)

 

In the next step, select the complete path for the .vmx file for the VM from the above step:

vm-support --vm /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/Test VMachine/Test VMachine.vmx

 

The VM logs will be generated and on completion it would output a path on the datastore as to where it is getting saved. 

In addition to the above, to customize the log bundle, i.e., suspending the VM, sending hung command etc., then the manifests could be listed using the below command.

vm-support --listmanifests

vm-support -h could be used to check for all the available options.

 

Workaround for exporting system logs for a virtual machine with "Send_NMI_To_Guest" option:

Use vm-support command-line to collect the System logs directly from the host:

 1. Log in to ESXi host through SSH or console as root.
Note: The virtual machine must be registered in the inventory.

 2. Run this command to list all registered and running virtual machines:

vm-support --listvms
vmdumper -l

 3. Run this command to Send NMI:

vm-support -a HungVM:Send_NMI_To_Guest --vm=/vmfs/volumes/Path/of/VMname.vmx

Refer KB: How to send NMI to Guest OS on ESXi 6.x or Later