"Host is disconnected" and "Host is connected" Events on the Virtual Machines

Products

• VMware vSphere ESXi 8.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• In the Events of the VMs, below events are found.
  
  
  
  
• The VMs will still be working normally.
• The ESXi Hosts may not get disconnect/reconnect in the vCenter.
• From /var/log/vmware/vpxd/vpxd.log of the vCenter, below log snippets are available:
  

  YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected
YYYY-MM-DDTHH:MM:SS vim.event.VmDisconnectedEvent info  <CLUSTER> ####### <CLUSTER> <VIRTUAL MACHINE> on host <ESXi HOST> in <CLUSTER> is disconnected

YYYY-MM-DDTHH:MM:SS info vpxd[######] [Originator@6876 sub=vmomi.soapStub[52] opID=HostSync-host-#####-######] SOAP request returned HTTP failure; <<io_obj p:0x00007fceb0609e28, h:####, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#####/vpxa>, method: getChanges; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH:MM:SS error vpxd[######] [Originator@6876 sub=Vmomi opID=HostSync-host-#####-########] Got vmacore exception when invoking VMOMI method; <</hgw/host-#####>, /vpxa>, vpxapi.VpxaService.getChanges, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
--> [context]zKq7AVECAQAAAI48ewEbdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUoBIEIebGlidm1vbWkuc28AAT9kIQHqkCEBXwohgpNFTAF2cHhkAII7i0wBgq2WTAGCtZdMAYJ3NEwBgm9iTAEB19oaA7MTDGxpYnZweGFwaS10eXBlcy5zbwCCCGUxAYLoxDEBglx5gQKCp4mBAoKbnIACgopzgQIABOw3ABdFOADFD1EEsI4AbGlicHRocmVhZC5zby4wAAXf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DDTHH:MM:SS warning vpxd[######] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-#####-########] Exception occurred during host sync; Host communication failed; [vim.HostSystem:host-#####,<ESXi Host>], e: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAQAAAI48ewEWdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUqBpe0cAXZweGQAgeb6KwGBBZlMAYF3NEwBgW9iTAEC19oabGlidm1vbWkuc28AA7MTDGxpYnZweGFwaS10eXBlcy5zbwCBCGUxAYHoxDEBgVx5gQKBp4mBAoGbnIACgYpzgQIABOw3ABdFOADFD1EEsI4AbGlicHRocmVhZC5zby4wAAXf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DDTHH:MM:SS info vpxd[######] [Originator@6876 sub=QuickStats opID=HostSync-host-#####-50f515cb] Host [vim.HostSystem:host-#####,<ESXi Host>] should not be polled
YYYY-MM-DDTHH:MM:SS warning vpxd[######] [Originator@6876 sub=MoHost opID=HostSync-host-#####-50f515cb] host [vim.HostSystem:host-#####,<ESXi Host>] connection state changed to NO_RESPONSE
YYYY-MM-DDTHH:MM:SS info vpxd[######] [Originator@6876 sub=QuickStats opID=HostSync-host-#####-50f515cb] Host [vim.HostSystem:host-#####,<ESXi Host>] should not be polled
YYYY-MM-DDTHH:MM:SS info vpxd[######] [Originator@6876 sub=vpxLro opID=3bacd923] [VpxLRO] -- BEGIN lro-10711729 -- session[########-####-####-####-############] -- vim.event.EventHistoryCollector.readNext -- 5244c22d-915a-d68c-8842-6ae08f131de2(52955860-0d64-ada3-0699-2be6066b729f)
  
  

• From /var/run/log/envoy.log of the ESXi Host, below log snippets are found:
  
  YYYY-MM-DDTHH:MM:SS In envoy[######]: "YYYY-MM-DDTHH:MM:SS warning envoy[#####] [Originator@#### sub=filter] [Tags: "ConnectionId":"######"] remote http connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SS In envoy[######]: "YYYY-MM-DDTHH:MM:SS warning envoy[#####] [Originator@#### sub=filter] [Tags: "ConnectionId":"######"] remote http connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SS In envoy[######]: "YYYY-MM-DDTHH:MM:SS warning envoy[#####] [Originator@#### sub=filter] [Tags: "ConnectionId":"######"] remote http connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SS In envoy[######]: "YYYY-MM-DDTHH:MM:SS warning envoy[#####] [Originator@#### sub=filter] [Tags: "ConnectionId":"######"] remote http connections exceed max allowed: 128"

Cause

• Scenario 1: 
  • The ESXi host's capacity for external HTTPS connections can be exhausted by an overload of requests originating from any single or multiple external devices targeting its management IP on TCP/443.
    
    Evidence in ESXi host's /var/run/log/envoy.log are found as follows:
    warning envoy[#######] [Originator@#### sub=filter] [Tags: "ConnectionId":"########"] remote https connections exceed max allowed: 128
    
    
• Scenario 2:
  • This limit is most commonly triggered by replication services (see Additional Information), backup solutions and vulnerability scanners.
    
    
• Scenario 3:
  • Multiple concurrent operations in SDDC Manager, driven by password rotations and LCM, utilize a shared connection path through vCenter to ESXi, resulting in an excessive load of HTTPS connection requests on the ESXi hosts.

Resolution

• Workaround:
  
  1. Restart the envoy service on the affected ESXi host to clear the sessions, using the below command on the affected host.

      /etc/init.d/envoy restart

  
  2. Identify the source of the HTTPS connections to the ESXi host Management IP, using below command and stop or limit these sessions from occurring.
  
    grep "<host-management-IP>:443" /var/run/log/envoy-access.* | cut -d' ' -f 15 | sort | uniq | cut -d ':' -f 1 | uniq -c
  
      Example O/p:
  
    grep "host-management-IP:443" /var/run/log/envoy-access.* | cut -d' ' -f 15 | sort | uniq | cut -d ':' -f 1 | uniq -c
    1485 ##.##.##.##
    1972 ##.##.##.##
  
  
  
• Permanent Solution:
  
  Involve Application Vendor to assist in ways to limit the number of HTTPS sessions, ensuring the sessions are closed when finished or prevent these sessions from recurring.

Additional Information

For additional information when replication is found to be causing this ESXi limit to be exceeded, see ESXi Hosts Show as "Not Responding" Due to Envoy Session Limits Exceeded by Replication Services