ESXi host repeatedly disconnects from the vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• ESXi hosts randomly get disconnected from the vCenter Server. A manual reconnection resolves the issue for a short period.
• The /var/log/vmware/vpxd/vpxd.log file on the vCenter Server shows repeated SSL handshake failures, as seen below:

YYYY-MM-DDTHH:MM:SS warning vpxd[####] [Originator@6876 sub=IO.Connection opID=####] Failed to SSL handshake; SSL(...), e: 167772294(certificate verify failed (SSL routines))
YYYY-MM-DDTHH:MM:SS warning vpxd[####] [Originator@6876 sub=HttpConnectionPool opID=####] Failed to get pooled connection; ... SSL Exception: Verification parameters:
--> PeerThumbprint: AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA
--> ExpectedThumbprint: BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB:BB
--> ExpectedPeerName: #######
--> * self-signed certificate in certificate chain

• TLS errors and certificates from intercepted traffic can be seen in the envoy-access and envoy logs:

/var/log/vmware/envoy-hgw/envoy-access.log

YYYY-MM-DDTHH:MM:SS info envoy[2580] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS POST /hgw/host-####/vpxa HTTP/1.1 526 upstream_reset_before_response_started{remote_connection_failure|TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end} UF 6606 1696 - 90 - - - - - <ESXi_IP>:443

/var/log/vmware/envoy-hgw/envoy.log

YYYY-MM-DDTHH:MM:SS info envoy-hgw[####] [Originator@6876 sub=connection] [Tags: "ConnectionId":"#######"] remote address:<IP_Address>,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
-----BEGIN CERTIFICATE-----
#################
-----END CERTIFICATE-----

• The presented certificate is issued by a third-party CA (Firewall, Proxy) rather than the expected ESXi Machine SSL certificate.

Environment

VMware vSphere ESXi 8.x

vCenter Server 8.x

Cause

• A network security device (firewall or SSL inspection proxy) intercepts vCenter-to-ESXi HTTPS traffic (TCP/443), decrypts it, and re-encrypts it using its own CA-issued certificate for the ESXi hostname.
• This replaces the original ESXi certificate (/etc/vmware/ssl/rui.crt) with a third-party one, causing a thumbprint mismatch during vCenter validation. The PeerThumbprint fails against the ExpectedThumbprint stored in vCenter, resulting in a handshake failure and host disconnection.
• To validate the issue, compare the fingerprint and the issuer of the original ESXi certificate with the intercepted certificate captured in /var/log/vmware/envoy-hgw/envoy.log:

From a shell session on the affected ESXi host:

openssl x509 -noout -text -in /etc/vmware/ssl/rui.crt -fingerprint | grep -E "Issuer|Fingerprint"

Against the intercepted certificate from envoy.log:

openssl x509 -noout -text -in /etc/vmware/ssl/intercepted-cert.crt -fingerprint | grep -E "Issuer|Fingerprint"

Resolution

Review any third-party products (such as a firewall doing SSL inspection) that handle network traffic between the ESXi host and vCenter Server, and disable SSL/TLS inspection.