"Num handlers exceed max allowed" error when connecting to ESXi/ESX host using vSphere Client

Products

VMware

Issue/Introduction

Symptoms:

Connecting to the ESXi/ESX host directly using the vSphere Client fails.
The host goes into the Not Responding state within vCenter Server.
In the hostd.log file on a ESXi 5.x host, you see entries similar to:

[2010-10-24 13:47:37.482 22083B90 warning 'Proxysvc'] Num handlers (128) exceed max allowed (128). Throttling accept
In the hostd.log file on a ESX 4.x host, you see entries similar to:

[2010-10-24 13:47:37.482 F63B3900 warning 'Proxysvc'] Num handlers (320) exceed max allowed (320). Throttling accept
[2010-10-24 13:47:37.499 F63B3900 verbose 'Proxysvc Req00440'] New proxy client SSL(TCP(local=192.168.10.230:58661, peer=192.168.10.231:443))
[2010-10-24 13:49:37.525 F566EB90 warning 'Proxysvc'] Num handlers (320) exceed max allowed (320). Throttling accept

Cause

This issue occurs when there are too many connections to the host and the host agent runs out of connection handlers.

Note: The maximum number of connection handlers is for ESXi 5.x is 128. The maximum number of connection handlers for ESX/ESXi 4.x is 320.

The host agent may run out of connection handlers if the firewall that implements SSL timeouts silently discards/drops connections. Any connection closure by the firewall must forward the connection state to both endpoints. If it does not forward the connection state to both endpoints, the connection handle remains open (TCP/ESTABLISHED), but is unusable.

Resolution

This issue is resolved in ESXi 6.5, available at VMware Downloads.

To work around this issue if you do not want to upgrade:

Run one of these commands to see the number of active sessions to the hostd agent:
- For ESXi 5.x, run the command:
  
  esxcli network ip connection list | grep ":443" | wc -l
  
  Note: If you run the command without | wc -l, the output provides information about the sessions. For example:
  
  tcp 0 0 127.0.0.1:443 127.0.0.1:52680 ESTABLISHED 5474098 rhttpproxy-work
  tcp 0 0 10.131.0.125:443 10.131.10.204:58591 ESTABLISHED 5474099 rhttpproxy-work
  tcp 0 0 10.131.0.125:443 10.131.10.204:49588 ESTABLISHED 5474099 rhttpproxy-work
  tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 9234 rhttpproxy-work
- For ESXi 4.1, run the command:
  
  esxcli network connection list | grep ":443" | wc -l
  
  Note: If you run the command without | wc -l, the output provides information about the sessions. For example:
  
  tcp 0 0 10.16.224.66:443 10.16.224.167:60226 ESTABLISHED 8353
  tcp 0 0 10.16.224.66:443 10.16.224.167:60225 ESTABLISHED 8350
  tcp 0 0 10.16.224.66:443 10.16.224.167:60224 ESTABLISHED 5736
  tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 4957
- For ESX 4.1, run the command:
  
  # netstat -anp | grep ":443" | wc -l
  
  Note: If you run the command without | wc -l, the output provides information about the sessions. For example:
  
  tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3146/vmware-hostd
  tcp 0 0 10.16.224.64:443 10.16.224.167:59982 ESTABLISHED 3146/vmware-hostd
  tcp 0 0 10.16.224.64:443 10.16.224.167:59980 FIN_WAIT2 -
  tcp 0 0 10.16.224.64:443 10.16.224.167:59990 ESTABLISHED 3146/vmware-hostd
  tcp 0 0 10.16.224.64:443 10.16.224.167:59991 ESTABLISHED 3146/vmware-hostd
  tcp 0 0 10.16.224.64:443 10.16.224.167:59989 ESTABLISHED 3146/vmware-hostd
If the output of the command in step 1 is close to 128 for ESXi 5.x or 320 for ESX/ESXi 4.x, restart management agents to close these connections. For more information, see Restarting the Management agents on an ESX or ESXi Server (1003490).

Note: To prevent this issue from recurring, you must also ensure that the firewall products used do not discard or drop idle connections silently.

Additional Information

vSphere Client を使用して ESXi/ESX ホストに接続すると「Num ハンドラが最大許容数を超えました (Num handlers exceed max allowed)」エラーが発生する
使用 vSphere Client 连接至 ESXi/ESX 主机时出现“处理程序的数量超过最大允许数量 (Num handlers exceed max allowed)”错误