ESXi 5.0/5.1 host disconnects during vApp operation or randomly in a vCloud environment
search cancel

ESXi 5.0/5.1 host disconnects during vApp operation or randomly in a vCloud environment

book

Article ID: 342825

calendar_today

Updated On:

Products

VMware Cloud Director VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • ESXi host disconnects from vCenter Server during a vApp operation such as Power On or Deployment.
  • vCloud Director exists in the affected environment
  • ESXi host disconnects from vCenter Server, even while in maintenance mode
  • The ESXi host can be pinged and a direct connection to the host using the vSphere Client is possible
  • The vpxd.log (located at %allusersprofile%\VMware\VMware VirtualCenter\Logs) contains entries similar to:

    2013-10-31T04:47:57.662Z [36550B90 error 'Default'] SSLStreamImpl::DoServerHandshake for SSL(TCP(local=172.18.99.16:443, peer=172.18.99.25:52287)): SSL_accept failed with Unexpected EOF
    2013-10-31T04:47:57.668Z [36550B90 warning 'Default'] SSL Handshake failed for stream TCP(local=172.18.99.16:443, peer=172.18.99.25:52287), error: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: Unexpected EOF)
    2013-10-31T04:47:57.672Z [35C03B90 warning 'Default'] Num handlers (128) exceed max allowed (128). Throttling accept
    2013-10-31T04:47:57.674Z [35C03B90 error 'Default'] SSLStreamImpl::DoServerHandshake for SSL(TCP(local=172.18.99.16:443, peer=172.18.99.25:52288)): SSL_accept failed with Unexpected EOF 2013-10-31T04:47:57.675Z [35C03B90 warning 'Default'] SSL Handshake failed for stream TCP(local=172.18.99.16:443, peer=172.18.99.25:52288), error: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: Unexpected EOF)
    2013-10-31T04:47:57.679Z [35C44B90 warning 'Default'] Num handlers (128) exceed max allowed (128). Throttling accept
    2013-10-31T04:47:57.680Z [35C44B90 error 'Default'] SSLStreamImpl::DoServerHandshake for SSL(TCP(local=172.18.99.16:443, peer=172.18.99.25:52289)): SSL_accept failed with Unexpected EOF
  • The log, rhttpproxy, (located at var/run/log) contains entries similar to:

    2013-10-31T04:47:57.369Z [36613B90 verbose 'Proxy Req 08780'] New proxy client SSL(TCP(local=172.18.99.16:443, peer=172.18.99.52:38260))
    2013-10-31T04:47:57.386Z [36389B90 verbose 'Proxy Req 08781'] New proxy client SSL(TCP(local=172.18.99.16:443, peer=172.18.99.52:52671))
    2013-10-31T04:47:57.402Z [35B81B90 verbose 'Proxy Req 08782'] New proxy client SSL(TCP(local=172.18.99.16:443, peer=172.18.99.50:49002))
    2013-10-31T04:47:57.418Z [35DCAB90 verbose 'Proxy Req 08783'] New proxy client SSL(TCP(local=172.18.99.16:443, peer=172.18.99.52:45872))
    2013-10-31T04:47:57.434Z [36799B90 verbose 'Proxy Req 08784'] New proxy client SSL(TCP(local=172.18.99.16:443, peer=172.18.99.52:44653))


Environment

VMware vSphere ESXi 5.1
VMware Cloud Director 5.1.x
VMware vSphere ESXi 5.0
VMware Cloud Director 1.5.x

Cause

This issue is caused when there are too many SSL connections from the vCloud Director to the hostd management agent to fetch thumbnails, causing host disconnects in vCenter Server. From the vpxd.log, you see the SSL handshake timeout. Because there are too many SSL connections, rhttpproxy did not accept a new SSL connection.

Resolution

This issue is resolved in vCloud Director 5.5.
To work around this issue when you cannot upgrade, use one of these methods:
Method 1:

Disable the thumbnail images in vCloud Director.

To disable the thumbnail images:
  1. Log in as Administrator to vCloud Director UI.
  2. Click Help and select About from the drop-down menu.
  3. Press Ctrl-Shift-U to enter Uber mode.
  4. A popup screen appears. (Use a different browser if this step is not successful.)
  5. Select the option Disable Thumbnail Images.
  6. Click OK.

    Note: If you are unable to enter Uber mode through the web browser, run the following db script.

    update config set VALUE ='1' where name ='thumbnailImagesOff'
    (if Oracle) Commit;

  7. Restart the vCloud Director cell.
  8. Log in to the vCloud Director as root and open a terminal window.
  9. Run command service vmware-vcd restart
Method 2:

Increase the SSL handshake timeout from the current 120s to 150s or 200s.

To increase the SSL handshake timeout:

  1. Open an SSH/PuTTY connection to the affected host(s) and edit /etc/vmware/rhttpproxy/config.xml

    <config>
    ........
    <vmacore>
    ........
    <ssl>
    ........
    <handshakeTimeoutMs>150000</handshakeTimeoutMs>
    ........
    </ssl>
    ........
    </vmacore>
    ........
    </config>

    1. Restart rhttpproxy with the command:

      /etc/init.d/rhttpproxy restart


    Additional Information


    vCloud 環境で、vApp の操作中または不定期に ESXi 5.0/5.1 ホストへの接続が切断される

    Impact/Risks:
    When using the second workaround, which requires a restart of the VCloud Director cell, downtime may be an issue.