Host Disconnects and Reconnects - Connection Pool exhausted, vpxd in busy state

Products

• VMware vCenter Server

Issue/Introduction

• Multiple hosts from several clusters on the workload domain get disconnected from vCenter.
• Random ESXi hosts disconnect & reconnect in the vCenter
• In the vpxd logs you see Connection Pool exhausted while the vpxd is in a busy state, vpxd log may rotate very fast 
• vCenter Inventory taking a long time to load
• Any activity like connecting or browsing inside the vCenter web is sluggish
• Search option in ui does not load the results
• Missed heartbeats exceeding > 700 seconds
• vpxd connection pool is exhausted, you might see below entries in log file /var/log/vmware/vpxd-svcs/vpxd-svcs.log :

<Date & Time> [tomcat-exec-30 ERROR com.vmware.cis.server.util.ConnectionManager opId=] VPXD Connection Pool is exhausted
<Date & Time> [tomcat-exec-30 ERROR com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl opId=] Error occurred checking permissions for [urn:vmomi:Folder:group-d1:<VC GUID>] with userName= EXAMPLE.COM\<username> groups= [EXAMPLE.COM\Users, EXAMPLE.COM\USER, EXAMPLE.COM\Domain Users] privileges= [System.Read]

 

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.0

Cause

Large DNS cache with stale entries results causing VPXD timeout attempting to find a working IP and connect to that working IP address for an ESX host.

Resolution

This issue has been resolved in VMware vCenter Server 8.0a. 

Workaround:

To workaround this issue in vCenter Server 6.x and 7.x, follow the below steps.

• SSH to the vCenter Server 
• Make the following configuration changes in /etc/dnsmasq.conf:
  
   - Enable "negative" caching
   - Increase cache size.
  
  Here is the sample dnsmasq.conf that you can copy on VC:

 cat /etc/dnsmasq.conf

listen-address=127.0.0.1
bind-interfaces
user=dnsmasq
group=dnsmasq

#no-negcache
no-hosts
log-queries=extra
log-facility=/var/log/vmware/dnsmasq.log
domain-needed
dns-forward-max=300
cache-size=16384
neg-ttl=86400

• Restart the 'dnsmasq' service after updating the dnsmasq.conf:
  
  systemctl restart dnsmasq