ESXi 5.0 host disconnects from vCenter Server when using the EMC Isilon NAS plug-in and reports the error: SSL Exception: The SSL handshake timed out local

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When using the Isilon NAS VAAI plug-in, you experience these symptoms:

• The ESXi 5.0 host becomes unresponsive after you restart the management services while installing custom certificates
• Running the vim-cmd vmsvc/getallvms command fails with the error:
  
  Failed to login: SSL Exception: The SSL handshake timed out local:
  
  
• In the vpxa.log file, you see entries similar to:
  
  2013-01-08T09:32:11.495Z [5E1C8B90 error 'Default'] [VpxaHalCnxHostagent] Failed to discover version: vim.fault.HttpFault
2013-01-08T09:32:11.495Z [5E1C8B90 warning 'Default'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2013-01-08T09:32:11.495Z [5E1C8B90 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 20003 ms
2013-01-08T09:32:31.497Z [5E123B90 verbose 'Default'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2013-01-08T09:32:51.500Z [5E186B90 error 'Default'] SSLStreamImpl::BIORead (07a3a550) timed out
2013-01-08T09:32:51.500Z [5E186B90 error 'Default'] SSLStreamImpl::DoClientHandshake (07a3a550) SSL_connect failed with BIO Error
2013-01-08T09:32:51.500Z [5E186B90 error 'HttpConnectionPool'] [ConnectComplete] Connect error SSL Exception: The SSL handshake timed out local: 127.0.0.1:61169 peer: 127.0.0.1:443.
2013-01-08T09:32:51.501Z [5E123B90 error 'Default'] [HttpUtil::ExecuteRequest] Error in sending request - SSL Exception: The SSL handshake timed out local: 127.0.0.1:61169 peer: 127.0.0.1:443.
2013-01-08T09:32:51.501Z [5E123B90 error 'Default'] [VpxaHalCnxHostagent] Failed to discover version: vim.fault.HttpFault
2013-01-08T09:32:51.501Z [5E123B90 warning 'Default'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2013-01-08T09:32:51.501Z [5E123B90 warning 'VpxProfiler'] VpxUtil_InvokeWithOpId [TotalTime] took 20003 ms
2013-01-08T09:33:11.501Z [FFF77B90 verbose 'Default'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2013-01-08T09:33:31.505Z [FFF56B90 error 'Default'] SSLStreamImpl::BIORead (07a3a550) timed out
  
  
• If you restart the host, hostd fails to initialize
• In the hostd.log file, you see a stack trace similar to:
  
  2013-01-08T10:29:03.859Z [FFACCA90 info 'Ovfmgrsvc'] Plugin started
2013-01-08T10:29:03.859Z [FFACCA90 info 'Partitionsvc'] Starting partitionsvc plugin
2013-01-08T10:29:03.859Z [FFACCA90 info 'Proxysvc'] Max proxy connections currently set to 128 , buffer size: 33000 bytes
2013-01-08T10:29:03.859Z [FFACCA90 info 'Proxysvc'] vmacore/ssl/useSSLCtxPool: true
*** In-memory logs end ***
error: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140A90F1:lib(20):func(169):reason(241))
backtrace:
[00] rip 1f23d163
[01] rip 1f09691e
[02] rip 1f02bb12
[03] rip 1f0b58ac
[04] rip 1f1ab3d8
[05] rip 1f1ac7e0
[06] rip 1f1b969a
[07] rip 1f13e7ef
[08] rip 1f140e67
[09] rip 1f14193c
[10] rip 05519e89
[11] rip 055147c5
[12] rip 1f04a42e
[13] rip 1f049734
[14] rip 05a1ca5a
[15] rip 05a1264d
[16] rip 05a221c5
[17] rip 10d4cefc
[18] rip 04ca6781

Environment

VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.0

Cause

This issue occurs due to an incompatibility between the EMC Isilon NAS VAAI plug-in and the SSL adapter in the ESXi management agent hostd.

Resolution

This is a known issue affecting ESXi 5.0.

To work around this issue, uninstall or remove the Isilon module.

To uninstall or remove the Isilon module:

1. Run this command to list the installed VIBs:
   
   # esxcli software vib list
   
   The Isilon module is usually at the top of the list.
   
   
2. Copy the name of the module.
3. Run this command to uninstall the VIB:

# esxcli software vib remove -n isilon_module_name
   
   
4. Reboot the host for the changes to take effect.

If the preceding steps do not resolve the issue, file a support request with VMware Support and note this Knowledge Base article ID (2042590) in the problem description. For more information, see How to Submit a Support Request.

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box