ESX hosts in a 9.x environment may disconnect from vCenter and reconnect within a minute

Products

• VMware vSphere ESXi

Issue/Introduction

In a vSphere 9.x environment, you may notice an ESX host enter a disconnected state and reconnect to the vCenter without any user intervention.

 

The vpxa logs may list entries similar to the following:

2025-1#-##T##:##:##.###Z In(166) Vpxa[#######]: [Originator@#### sub=vpxLro opID=########-##-##] [VpxLRO] -- BEGIN lro-####### -- vsanSystem -- vim.host.VsanSystem.fetchVsanSharedSecret -- ########-####-####-####-############
2025-1#-##T##:##:##.###Z In(166) Vpxa[#######]: [Originator@#### sub=IO.Http opID=WFU-########] Set user agent error; state: 3, SSL(<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:17, <TCP '127.0.0.1 : 54959'>, <TCP '127.0.0.1 : 443'>>), N7Vmacore4Http24MalformedHeaderExceptionE(Server closed connection after 0 response bytes read)

 

The hostd logs may list entries similar to the following:

2025-1#-##T##:##:##.###Z Er(163) Vpxa[#######]: [Originator@#### sub=VpxaCnxHostd opID=WFU-########] Fatal error while processing updates: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
2025-1#-##T##:##:##.###Z Er(163) Vpxa[#######]: --> )
2025-1#-##T##:##:##.###Z Er(163) Vpxa[#######]: --> [context]<context_ID>==[/context]
2025-1#-##T##:##:##.###Z In(166) Vpxa[#######]: [Originator@#### sub=vpxaInvtHost opID=WFU-########] ServerId has been changed from ###### to 0
2025-1#-##T##:##:##.###Z Er(163) Vpxa[#######]: [Originator@#### sub=vpxaInvtHostCnx opID=WFU-########] Can't connect to hostd. Shutting down...
2025-1#-##T##:##:##.###Z In(166) Vpxa[#######]: [Originator@#### sub=Default opID=WFU-########] [Vpxa] Shutting down now

 

The envoy logs may list entries similar to the following:

2025-1#-##T##:##:##.###Z In(166) envoy-access[#######]: POST /sdk 0 downstream_remote_disconnect DC 942 0 - 1931 - - 127.0.0.1:52127 HTTP/1.1 TLSv1.3 127.0.0.1:443 0 1068 127.0.0.1:58842 HTTP/2 - 127.0.0.1:8307 "########-##-##" "FetchVsanSharedSecret"

Environment

• vSphere 9.x

Cause

This issue is a rare occurrence caused by a decrease of the idle timeout period for the envoy service.  The envoy idle timeout is currently set to 15 minutes in ESX 9.x.  Previous releases utilized a higher idle timeout value.

Resolution

The product development team is aware of this issue, and there is a plan to address the default idle connection timeouts of envoy in a future 9.x release.