In Nutanix environment, ESXi Host Intermittently Disconnects Due to Connection Limit Exhaustion

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• The ESXi host in vCenter intermittently shows as disconnected for a few seconds before reconnecting var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDTHH:MM:SS error vpxd[#####] [Originator@6876 sub=VmProv] Failed to set config settings on host; h: <Host FQDN>, key: 'esx/health/mobility_network_membership', v: '{{  : { destination_hosts :[ {{  : { host_name : <Host FQDN>, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 :########## , vmk_name : vmk2, } }} , ] , } }} , {{  : { host_name : <Host FQDN>
, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 : ##########, vmk_name : vmk2, } }} , ] , } }} , {{  : , provisioning_addresses : [ ] , vmotion_addresses : [ {{
: { ip_v4 : ##########, vmk_name : vmk2, } }} , ] , } }} , {{  : { host_name : <Host FQDN>, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 ##########: , vmk_name : vmk2, } }} , ] , } }
--> "host_name": "<Host FQDN>",

• In the var/log/vmware/vpxd/vpxd.log hearbeat is missing:

YYYY-MM-DDTHH:MM:SS warning vpxd[######] [Originator@6876 sub=QuickStats opID=PollQuickStatsLoop-#######] Error returned from calling FetchQuickStats for [vim.HostSystem:host-#####,<Host FQDN>]: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
YYYY-MM-DDTHH:MM:SS warning vpxd[######] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-####-########] Connection not alive due to missing heartbeats; [vim.HostSystem:host-######,<Host FQDN>], cnx: ############################

• Host /var/run/log/envoy/envoy.log shows https connections have exceeded the max allowed limit:
   
  

  YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] remote https connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] closing connection TCP<Remote entity-IP:<>, ESXi-IP:<>>"

Cause

This issue may occur when a user remains on Nutanix Prism page (like the Hardware page) for an extended period, causing persistent, unclosed connections to the ESXi host.

Resolution

• Establish an SSH session to the ESXi host as a root user:

• Run the following command:

  esxcli network ip connection list | grep tcp | grep envoy | grep ESTAB | wc -l



  Below is an expected output of the above command:
  
  

• If the output returns more than 128 established connections, please engage the Nutanix team for further investigation.

Additional Information

Nutanix Article on the issue: ESXi Host Intermittent Disconnection Due to Connection Limit Exhaustion