Adding or Re-connecting ESXi host to vCenter server may fail with "Cannot contact the specified host" due to MTU mismatch

Products

• VMware vSphere ESXi

Issue/Introduction

• Adding/Re-connecting host on vCenter server fails with "Cannot contact the specified host (<hostname>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
• The same host is accessible via ESXi Host Client
• /var/log/vmware/vpxd/vpxd.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS error vpxd[PID] [Originator@6876 sub=vpxCrypt opID=HeartbeatModuleStart-2e1bc7b0] [bool VpxPublicKey::Verify(const EVP_MD*, const unsigned char*, size_t, const unsigned char*, size_t) const] ERR error:1C880004:Provider routines::RSA lib
YYYY-MM-DDTHH:MM:SS warning vpxd[PID] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-2e1bc7b0] Failed to verify heartbeat signature; [vim.HostSystem:host-<Host ID>,<ESXi hostname>], cert: <ESXi Thumbprint>, signature: <>, msg: {srv: 548531, gen: 12906, ct:

• Performing a certificate check from the VCSA to ESXi is connected but does not display the certificate

openssl s_client -connect <ESXi IP/FQDN>:443
CONNECTED(00000003)

Cause

This issue stems from a Layer 2 MTU (Maximum Transmission Unit) mismatch, where the physical switch port is unable to accommodate the payload size of the ESXi host's MTU configuration, resulting in dropped frames.

Analyzing network packet capture from the host during the re-connect using Wireshark displays TCP Retransmission with packet length as Len=1386

YYYY-MM-DD HH:MM:SS    <VCSA IP>    <ESXi IP>    TCP         74       42232 → 443 [SYN] Seq=0 Win=29200 Len=0 MSS=1398 SACK_PERM TSval=######### TSecr=0 WS=256
YYYY-MM-DD HH:MM:SS    <ESXi IP>    <VCSA IP>    TCP         74       443 → 42232 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1398 WS=512 SACK_PERM TSval=######### TSecr=#########
YYYY-MM-DD HH:MM:SS    <VCSA IP>    <ESXi IP>    TCP         66       42232 → 443 [ACK] Seq=1 Ack=1 Win=#### Len=0 TSval=1657804849 TSecr=#########
YYYY-MM-DD HH:MM:SS    <VCSA IP>    <ESXi IP>    TLSv1.2     249      Client Hello (SNI=<ESXi hostname>)
YYYY-MM-DD HH:MM:SS    <ESXi IP>    <VCSA IP>    TLSv1.2     6006     Server Hello, Certificate, Server Key Exchange, Server Hello Done
YYYY-MM-DD HH:MM:SS    <VCSA IP>    <ESXi IP>    TCP         78       [TCP Dup ACK 72#1] 42232 → 443 [ACK] Seq=184 Ack=1 Win=30464 Len=0 TSval=######### TSecr=######### SLE=#### SRE=####
YYYY-MM-DD HH:MM:SS    <ESXi IP>    <VCSA IP>    TCP         1452     [TCP Retransmission] 443 → 42232 [ACK] Seq=1 Ack=184 Win=66048 Len=1386 TSval=######### TSecr=#########
YYYY-MM-DD HH:MM:SS    <ESXi IP>    <VCSA IP>    TCP         1452     [TCP Retransmission] 443 → 42232 [ACK] Seq=1 Ack=184 Win=66048 Len=1386 TSval=######### TSecr=#########

Resolution

Engage the internal network team to isolate and address the MTU misconfiguration.

As a workaround, modify the MTU on the ESXi host as per the Length reported in the packet capture. Refer to vSphere Standard Switch Properties

Additional Information

Troubleshooting an ESXi host in a "not responding"/"disconnected" state 

Troubleshooting ESXi Host in Disconnected State and Unable to Connect to vCenter