Nutanix Hosts flip between Not Responding and Connected status in vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host managed by vCenter becomes unresponsive and shows disconnected in vSphere client. The virtual machines on this host are still running, but without vCenter connectivity, they cannot be migrated to other hosts for maintenance or troubleshooting purposes.
• In ESXi, /var/run/log/envoy.log shows https connections have exceeded the max allowed limit: 
  

  YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] remote https connections exceed max allowed: 128"

  YYYY-MM-DDTHH:MM:SS:MS In(166) envoy[41277035]: "YYYY-MM-DDTHH:MM:SS:MSZ warning envoy[41277051] [Originator@6876 sub=filter] [C292324] closing connection TCP<Remote entity-IP:55252, ESXi-IP:443>"

• Restarting envoy service on the ESXi host temporarily fixes the issue.

• localcli network ip connection list returns the majority of the ESTABLISHED connections by Nutanix CVMs exhausting the max allowed limit. 

• vCenter connections get queued up in the TIME_WAIT state.

Resolution

This is a known issue with Nutanix. Please engage Nutanix Support for further assistance. 

Additional Information

Nutanix KB : https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0VO0000008ejR0AQ