ESXI Hosts enters "not responding" state after adding it to vCenter

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• ESXI goes into not responding state after adding/re-adding it back to the vCenter.
• Connection exceed between the replication server and ESXi.
  • In "/var/run/log/envoy.log" we see below entries:
    

    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21972] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21972] closing connection TCP<######:54892, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235958] [Originator@6876 sub=filter] [C21973] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235958] [Originator@6876 sub=filter] [C21973] closing connection TCP<######:54898, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21974] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21974] closing connection TCP<######:54910, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21975] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21975] closing connection TCP<######:54914, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21976] remote https connections exceed max allowed: 128"

     

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vSphere ESXi 7.X
• VMware vSphere ESXi 8.X

Cause

Authentication issue between the hms and hbrsrvuw in ESXi. This can randomly affect any ESXi hosts in the vCenter.

Resolution

Workaround : 

• When using vSphere Replication, all hosts within the vCenter Server inventory are registered automatically. If you want to exclude hosts or hosts under a given cluster or datacenter from registration in vSphere Replication, you can tag them with the com.vmware.vr.disallowedHost tag Installing and Setting Up vSphere Replication

Fix:

1. This issue is fixed in vSphere Replication 9.0.1

Additional Information

Read vSphere Replication sending large number of HTTPS requests to envoy proxy causing hostd to crash - remote https connections exceed max allowed for more information on the issue