Generating ESXi log bundle from vCenter Server will time out with the error: Operation timed out

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Collecting an ESXi log bundle from vCenter Server fails with the error:
  
  Operation timed out.
  
• The issue is observed when collecting logs from vSphere Client and vSphere Web Client.
• The issue is not observed when collecting logs from command line of the ESXi host.
• The /var/log/hostd.log file shows entries similar to:

YYYY-MM-DDT<time> [3C2E6B90 verbose 'Solo.CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command started.</time>

YYYY-MM-DDT<time> [3CE80B90 error 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Timeout on stdout! Terminating!</time>

YYYY-MM-DDT<time> [3CA40B90 error 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Error writing to response: N7Vmacore11IOExceptionE(IO error: System exception while trasmitting HTTP Response:</time>

--> error id = 32

--> )

YYYY-MM-DDT<time> [3CA40B90 verbose 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] CGI command killed!</time>

YYYY-MM-DDT<time> [3CEC1B90 verbose 'Solo.CgiResponseWriter (/cgi-bin/vm-support.cgi)'] Close</time>

YYYY-MM-DDT<time> [3CEC1B90 verbose 'Solo.CgiSysCommandHandler (/cgi-bin/vm-support.cgi)'] CGI command finished.</time>

• The action.log file in the root directory of the log bundle when generated from the command line shows an entry similar to:

YYYY-MM-DDT<time>: run IGNORE_ERROR,TIMEOUT,RELOCATE /usr/lib/vmware/vm-support/bin/storageHostProfiles.sh (/etc/vmware/vm-support/hostProfiles.mfx) took 1805.000 sec</time>

• The error.log file in the root directory of the log bundle when generated from the command line shows an entry similar to:

cmd /usr/lib/vmware/vm-support/bin/storageHostProfiles.sh timed out after 1800 seconds due to lack of progress in last 900 seconds (0 bytes read)

Environment

VMware vCenter Server 5.5.x
VMware vCenter Server 5.1.x
VMware vSphere ESXi 6.0
VMware vCenter Server 6.0.x
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1

Cause

This issue occurs when one of the sub-routines of collecting the log bundle (storageHostProfiles.sh script) runs for over 15 minutes without output, which causes a CgiResponseWriter timeout.

This script gathers Host Profile data for the host and prints PSA/NMP and VVOL related profiles by collecting information from the LUNs on the host. When there are many LUNs attached to the ESXi host, this can cause the script to reach the timeout limit. The timeout is observed in vSphere Client and vSphere Web Client because the command is supervised by hostd, and a timer is used to control the time spent on the remote session.

Resolution

Currently, there is no resolution.

To work around this issue, collect the logs from the command line of the ESXi host with the command:

# vm-support

Note: Though storageHostProfiles.sh is used when gathering logs from the command line of the ESXi host, there is no timer for the vm-support command when run directly from the ESXi command line, which allows the operation to complete.

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

vCenter Server から ESXi ログ バンドルを収集すると次のエラーでタイムアウトになる：操作はタイム アウトしました。
从 vCenter Server 生成 ESXi 日志包超时并显示错误：操作超时 (Operation timed out)