ESXI Hosts enters "not responding" state after adding it to vCenter

Products

• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• ESXI goes into not responding state after adding/re-adding it back to the vcenter.
• Connection exceed between the replication server and ESXI 

/var/run/log/envoy.log 

YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.607Z warning envoy[10235957] [Originator@6876 sub=filter] [C21972] remote https connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.607Z warning envoy[10235957] [Originator@6876 sub=filter] [C21972] closing connection TCP<######:54892, ######:443>"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.636Z warning envoy[10235958] [Originator@6876 sub=filter] [C21973] remote https connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.636Z warning envoy[10235958] [Originator@6876 sub=filter] [C21973] closing connection TCP<######:54898, ######:443>"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.640Z warning envoy[10235957] [Originator@6876 sub=filter] [C21974] remote https connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.640Z warning envoy[10235957] [Originator@6876 sub=filter] [C21974] closing connection TCP<######:54910, ######:443>"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.742Z warning envoy[10235957] [Originator@6876 sub=filter] [C21975] remote https connections exceed max allowed: 128"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.742Z warning envoy[10235957] [Originator@6876 sub=filter] [C21975] closing connection TCP<######:54914, ######:443>"
YYYY-MM-DDTHH:MM:SSZ In(166) envoy[10235567]: "2024-10-24T11:36:29.742Z warning envoy[10235957] [Originator@6876 sub=filter] [C21976] remote https connections exceed max allowed: 128"

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

VMware vSphere ESXi 7.X

VMware vSphere ESXi 8.X

Cause

Authentication issue between the hms and hbrsrvuw in ESXi. This can randomly affect any ESXi host in the vCenter.

Resolution

Workaround : 

When using vSphere Replication, all hosts within the vCenter Server inventory are registered automatically. If you want to exclude hosts or hosts under a given cluster or datacenter from registration in vSphere Replication, you can tag them with the com.vmware.vr.disallowedHost tag Installing and Setting Up vSphere Replication

Fix:

1. Upgrade the vSphere replication to version 9.0.0.1 and above

2  Restart HMS service on the vSphere replication Start, Stop, and Restart vSphere Replication Appliance Services