Generating ESXi log bundle from vCenter Server will time out with the error: Operation timed out

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Collecting an ESXi log bundle from vCenter Server fails with the error:
  
  Operation timed out.
  
• The issue is observed when collecting logs from vSphere Client and vSphere Web Client.
• The issue is not observed when collecting logs from command line of the ESXi host.
• The /var/log/hostd.log file shows entries similar to:

YYYY-MM-DDT<time> [3C2E6B90 verbose 'Solo.CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command started.</time>

YYYY-MM-DDT<time> [3CE80B90 error 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Timeout on stdout! Terminating!</time>

YYYY-MM-DDT<time> [3CA40B90 error 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Error writing to response: N7Vmacore11IOExceptionE(IO error: System exception while trasmitting HTTP Response:</time>

--> error id = 32

--> )

YYYY-MM-DDT<time> [3CA40B90 verbose 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] CGI command killed!</time>

YYYY-MM-DDT<time> [3CEC1B90 verbose 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Close</time>

YYYY-MM-DDT<time> [3CEC1B90 verbose 'Solo.CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command finished.</time>

• The action.log file in the root directory of the log bundle when generated from the command line shows an entry similar to:

YYYY-MM-DDT<time>: run IGNORE_ERROR,TIMEOUT,RELOCATE /usr/lib/vmware/vm-support/bin/storageHostProfiles.sh (/etc/vmware/vm-support/hostProfiles.mfx) took 1805.000 sec</time>

• The error.log file in the root directory of the log bundle when generated from the command line shows an entry similar to:

cmd /usr/lib/vmware/vm-support/bin/storageHostProfiles.sh timed out after 1800 seconds due to lack of progress in last 900 seconds (0 bytes read)

Environment

VMware vCenter Server
VMware vSphere ESXi

Cause

This issue occurs when one of the sub-routines of collecting the log bundle (storageHostProfiles.sh script) runs for over 15 minutes without output, which causes a CgiResponseWriter timeout.

This script gathers Host Profile data for the host and prints PSA/NMP and VVOL related profiles by collecting information from the LUNs on the host. When there are many LUNs attached to the ESXi host, this can cause the script to reach the timeout limit. The timeout is observed in vSphere Client and vSphere Web Client because the command is supervised by hostd, and a timer is used to control the time spent on the remote session.

Resolution

Currently, there is no resolution.

To work around this issue, collect the logs from the command line of the ESXi host with the command:

# vm-support

Note: Though storageHostProfiles.sh is used when gathering logs from the command line of the ESXi host, there is no timer for the vm-support command when run directly from the ESXi command line, which allows the operation to complete.

Additional Information