Alarms about the host connection state changing from green to red frequently occur

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

Host connection state alarms change from green to red frequently (for example, daily).
ESXi/ESX hosts go into a state of Not Responding in vCenter Server then respond again after a few seconds.
This issue recurs after alarms are reset.
The vCenter Server vpxd log files contain entries similar to:

[YYYY-MM-DD 02:26:02.314 58326B90 error 'App'] SSLStreamImpl::BIORead (590AE458) timed out
YYYY-MM-DD 02:26:02.315 17B2DB90 error 'App'] SSLStreamImpl::BIORead (591EF770) timed out
YYYY-MM-DD 02:26:02.315 58326B90 info 'TCP'] close(68)
YYYY-MM-DD 02:26:02.315 17B2DB90 info 'TCP'] close(66)
[YYYY-MM-DD 02:26:02.315 58326B90 error 'App'] SSLStreamImpl::DoServerHandshake (590AE458) SSL_accept failed with BIO Error
[YYYY-MM-DD 02:26:02.316 17B2DB90 error 'App'] SSLStreamImpl::DoServerHandshake (591EF770) SSL_accept failed with BIO Error
[YYYY-MM-DD 02:26:02.316 58326B90 warning 'Proxysvc'] SSL Handshake timedout for stream 10.172.16.17, blacklisting it for 3000 ms

Note: The vpxd logs are located at %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs, which translates to:

C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs in Windows 2003
C:\ProgramData\VMware\VMware VirtualCenter\Logs in Windows 2008

The hostd.log file (located at /var/log/vmware/ in ESX/ESXi 4.x and /var/log in ESXi 5.x) contain entries similar to:

[YYYY-MM-DD 18:26:45.580 00948 error 'App'] SSLStreamImpl::DoClientHandshake (1B2AB0D8) SSL_connect failed with Unexpected EOF

Environment

VMware ESXi 3.5.x Embedded
VMware vCenter Server 6.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server 5.0.x
VMware ESXi 3.5.x Installable
VMware ESX Server 3.5.x
VMware ESX Server 3.0.x
VMware vCenter Server 5.1.x
VMware vCenter Server 4.0.x
VMware ESXi 4.1.x Installable
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESX 4.0.x
VMware vCenter Server 4.1.x
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1

Cause

This issue occurs on a busy network when the Secure Socket Layer (SSL) timeout value is too short or if the handshakeTimeoutMs value in the /etc/vmware/hostd/config.xml file is set too low.

Resolution

These alarms occur even if the host is working correctly.

To work around this issue, increase the value of handshakeTimeoutMs:

Connect to your ESXi/ESX host via a remote Kernel-based Virtual Machine (KVM) or Secure Shell (SSH) session or directly as root.
Open the /etc/vmware/hostd/config.xml file in a text editor.

Note: For ESXi 5.x and ESXi 6.x open the /etc/vmware/rhttpproxy/config.xml
Enter <handshakeTimeoutMs> 120000 </handshakeTimeoutMs> between the <ssl> tags in the vmacore section. This parameter is in milliseconds. For example, 120000 millseconds = 2 minutes.

ESXi 5.x

ESXi 6.x:　Add <handshakeTimeoutMs> entity

    <vmacore>
    ...
     <ssl>
         <doVersionCheck> false </doVersionCheck>
         <useCompression>true</useCompression>
         <libraryPath>/lib/</libraryPath>
         <handshakeTimeoutMs>120000</handshakeTimeoutMs>
     </ssl>
     ...
    </vmacore>

Save and close the file.
Restart the Management agents. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).
To confirm that the tags have taken effect, check the /var/log/vmware/hostd.log file for this message

[YYYY-MM-DD HH:MM:SS.SSS F66D76D0 info 'App'] Vmacore::InitSSL: doVersionCheck = false, handshakeTimeoutUs = 120000000

Note: For ESXi 5.x and 6.x, check the /var/log/vmware/rhttpproxy.log

SSL Async Handshake Timeout : Read timeout after approximately 120000 ms. Closing stream <SSL(<io_obj p:0x35516694, h:16, <TCP 'XXX.XXX.XXX.XXX:443'>, <TCP 'XXX.XXX.XXX.XXX:51373'>>)>..

Note: The handshakeTimeoutUs value is shown in microseconds

Additional Information

For translated versions of this article, see: