ESXi host repeatedly disconnects from vCenter

Products

• VMware vCenter Server

Issue/Introduction

ESXi hosts will randomly disconnect from its vCenter. A manual reconnection resolves the issue for a time

/var/log/vmware/vpxd/vpxd.log shows repeated SSL handshake failures:

XXXX-XX-XXTXX:XX:XX.XXX+XX:XX warning vpxd[XXXXX] [Originator@6876 sub=IO.Connection opID=XXXX] Failed to SSL handshake; SSL(...), e: 167772294(certificate verify failed (SSL routines))
XXXX-XX-XXTXX:XX:XX.XXX+XX:XX warning vpxd[XXXXX] [Originator@6876 sub=HttpConnectionPool-XXXX opID=XXXX] Failed to get pooled connection; ... SSL Exception: Verification parameters:
--> PeerThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
--> ExpectedThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
--> ExpectedPeerName: XXXX
--> * self-signed certificate in certificate chain

/var/log/vmware/envoy-hgw/envoy.log captures TLS errors and certificate from intercepted traffic :

XXXX-XX-XXTXX:XX:XX.XXXZ info envoy-hgw[XXXX] [Originator@6876 sub=connection] [Tags: "ConnectionId":"XXXXXXX"] remote address:XXX.XXX.XXX.XXX:XXX,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXX
-----END CERTIFICATE-----

The presented certificate is issued by a third-party CA (Firewall,Proxy), not the expected ESXi Machine SSL certificate.

Environment

vCenter 8.x, 7.x

ESXi 8.x, 7.x

Cause

A network security device (firewall or SSL inspection proxy) intercepts vCenter-to-ESXi HTTPS traffic (TCP/443), decrypts it, and re-encrypts using its own CA-issued certificate for the ESXi hostname.

This replaces the original ESXi certificate (/etc/vmware/ssl/rui.crt) with a third-party one, causing thumbprint mismatch during vCenter validation. The PeerThumbprint fails against the ExpectedThumbprint stored in vCenter, resulting in handshake failure and host disconnection.

To validate the issue compare the fingerprint and the issuer of the original ESXi certificate :

from shell session on the affected ESXi

openssl x509 -noout -text -in /etc/vmware/ssl/rui.crt -fingerprint | grep -E "Issuer|Fingerprint"

with the certificate captured in /var/log/vmware/envoy-hgw/envoy.log

openssl x509 -noout -text -in /etc/vmware/ssl/intercepted-cert.crt -fingerprint | grep -E "Issuer|Fingerprint"

Resolution

Review any 3rd party products (firewall doing SSL inspection) that handles network traffic between the ESXi and vCenter and  disable SSL/TLS inspection.