Alarms about the host connection state changing from green to red frequently occur

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Host connection state alarms change from green to red frequently (for example, daily).
ESXi/ESX hosts go into a state of Not Responding in vCenter Server, then respond again after a few seconds.
This issue recurs after alarms are reset.
The vCenter Server vpxd log files contain entries similar to:

[YYYY-MM-DD 02:26:02.314 58326B90 error 'App'] SSLStreamImpl::BIORead (590AE458) timed out
[YYYY-MM-DD 02:26:02.315 17B2DB90 error 'App'] SSLStreamImpl::BIORead (591EF770) timed out
[YYYY-MM-DD 02:26:02.315 58326B90 info 'TCP'] close(68)
[YYYY-MM-DD 02:26:02.315 17B2DB90 info 'TCP'] close(66)
[YYYY-MM-DD 02:26:02.315 58326B90 error 'App'] SSLStreamImpl::DoServerHandshake (590AE458) SSL_accept failed with BIO Error
[YYYY-MM-DD 02:26:02.316 17B2DB90 error 'App'] SSLStreamImpl::DoServerHandshake (591EF770) SSL_accept failed with BIO Error
[YYYY-MM-DD 02:26:02.316 58326B90 warning 'Proxysvc'] SSL Handshake timedout for stream 10.172.16.17, blacklisting it for 3000 ms

The vpxd.log can be found in the following sections:

Appliance vCenter

/var/log/vmware/vpxd/vpxd.log

Windows vCenter

The vpxd logs are located at %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs, which translates to:

C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs in Windows 2003

C:\ProgramData\VMware\VMware VirtualCenter\Logs in Windows 2008

ESXi - /var/log/hostd.log contain entries similar to:

[YYYY-MM-DD 18:26:45.580 00948 error 'App'] SSLStreamImpl::DoClientHandshake (1B2AB0D8) SSL_connect failed with Unexpected EOF

Environment

ESXi 6.x
vCenter Server 6.x

Cause

This issue occurs on a busy network when the Secure Socket Layer (SSL) timeout value is too short or if the handshakeTimeoutMs value in the /etc/vmware/hostd/config.xml file is set too low.

Resolution

These alarms occur even if the host is working correctly.

Workaround

Increase the value of handshakeTimeoutMs.

Connect to your ESXi/ESX host via a remote Kernel-based Virtual Machine (KVM) or Secure Shell (SSH) session or directly as root.
Open the /etc/vmware/rhttpproxy/config.xml file in a text editor.
Enter <handshakeTimeoutMs> 120000 </handshakeTimeoutMs> between the <ssl> tags in the vmacore section. This parameter is in milliseconds. For example, 120000 millseconds = 2 minutes.

ESXi 6.x:　

Add <handshakeTimeoutMs> entity

    <vmacore>
    ...
     <ssl>
         <doVersionCheck> false </doVersionCheck>
         <useCompression>true</useCompression>
         <libraryPath>/lib/</libraryPath>
         <handshakeTimeoutMs>120000</handshakeTimeoutMs>
     </ssl>
     ...
    </vmacore>

Save and close the file.
Restart the Management agents. For more information, see Restarting the Management agents on an ESXi or ESX host.
To confirm that the tags have taken effect, check the /var/log/vmware/hostd.log file for this message.

[YYYY-MM-DD HH:MM:SS.SSS F66D76D0 info 'App'] Vmacore::InitSSL: doVersionCheck = false, handshakeTimeoutUs = 120000000
Check the ESXi - /var/log/vmware/rhttpproxy.log for:

SSL Async Handshake Timeout : Read timeout after approximately 120000 ms. Closing stream <SSL(<io_obj p:0x35516694, h:16, <TCP '#.#.#.#:443'>, <TCP '#.#.#.#:51373'>>)>..

Note: The handshakeTimeoutUs value is shown in microseconds.