Nutanix Hosts flip between Not Responding and Connected status in vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host managed by vCenter becomes unresponsive and shows disconnected in vSphere client. The virtual machines on this host are still running, but without vCenter connectivity, they cannot be migrated to other hosts for maintenance or troubleshooting purposes.
• In ESXi, /var/run/log/envoy.log shows https connections have exceeded the max allowed limit: 
  

  YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] remote https connections exceed max allowed: 128"

  YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] closing connection TCP<Remote entity-IP:55252, ESXi-IP:443>"

• Restarting envoy service on the ESXi host temporarily fixes the issue.

• localcli network ip connection list returns the majority of the ESTABLISHED connections by Nutanix CVMs exhausting the max allowed limit. 

• vCenter connections get queued up in the TIME_WAIT state.

Cause

This issue happens if the user stays a long time on the hardware page in the Nutanix Prism UI, making periodic calls to the ESXi host in order to determine whether the host is MM or not. But somehow it doesn't close the connection, and it's leading up to hitting the limit. 

Resolution

This is a known issue with Nutanix. Please engage Nutanix Support for further assistance. 

Nutanix KB: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0VO0000008ejR0AQ