ESXI Hosts enters "not responding" state after adding it to vCenter

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• ESXI goes into not responding state after adding/re-adding it back to the vCenter.
• Connection exceed between the replication server and ESXi.
  • In "/var/run/log/envoy.log" we see below entries:
    

    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21972] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21972] closing connection TCP<######:54892, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235958] [Originator@6876 sub=filter] [C21973] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235958] [Originator@6876 sub=filter] [C21973] closing connection TCP<######:54898, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21974] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21974] closing connection TCP<######:54910, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21975] remote https connections exceed max allowed: 128"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21975] closing connection TCP<######:54914, ######:443>"
    YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "YYYY-MM-DDTHH:MM:SSZ warning envoy[10235957] [Originator@6876 sub=filter] [C21976] remote https connections exceed max allowed: 128"

     

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

VMware vSphere ESXi 7.X

VMware vSphere ESXi 8.X

Cause

Authentication issue between the hms and hbrsrvuw in ESXi. This can randomly affect any ESXi hosts in the vCenter.

Resolution

Workaround : 

• When using vSphere Replication, all hosts within the vCenter Server inventory are registered automatically. If you want to exclude hosts or hosts under a given cluster or datacenter from registration in vSphere Replication, you can tag them with the com.vmware.vr.disallowedHost tag Installing and Setting Up vSphere Replication

Fix:

1. VMware is aware of the issue and the issue will be fixed in upcoming vSphere Replication build.
2. Restart HMS service on the vSphere replication Start, Stop, and Restart vSphere Replication Appliance Services