Adding new or disconnected ESXi hosts to vCenter fails with error "Cannot contact host"

Products

• VMware vCenter Server

Issue/Introduction

Connecting disconnected ESXi hosts would take long and eventually fail with "Cannot contact host FQDN/IP".

• ESXi hosts accessible via SSH or GUI.
• ESXi hosts removed from the vCenter inventory will not add back to the vCenter.
• New ESXi hosts fails to add to vCenter.
• Ports 443 & 902 are open and connecting from vCenter to ESXi host.
• MTU check with following command fails:
  
  From vCenter:
  ping -M do -s 1472 ESXI_Host_IP
  From  ESXi host:
  ping -d -s 1472 vCenter_IP

Environment

VMware vCenter Server 8.0.x
VMware vSphere ESXi 8.0.x

Cause

MTU mismatch in the end to end communication between vCenter and ESXi hosts.

Resolution

Reach out to your internal networking team to configure same MTU across the virtual (vCenter/ESXi/vSwitch/vmkernel) & physical network (router/switch).

Workaround:

1. Take an offline snapshot of standalone vCenter OR of all vCenter in ELM.
2.  Edit networking file "/etc/systemd/network/10-eth0.network"
   vCenter is set to 1500 MTU byte size by default unless explicitly defined in file 10-eth0.network
3. Determine the acceptable MTU byte packet size to be defined in network file by randomly pinging with different MTU byte packet.
   Example:
   Try pinging with MTU byte size 1400 and then gradually reduce or increase to determine what packet size transmits using following commands:
   
   • From vCenter:
     ping -M do -s 1472 ESXI_Host_IP
     From  ESXi host:
     ping -d -s 1472 vCenter_IP
4. Take a backup of the file using the below command:
   cp /etc/systemd/network/10-eth0.network /etc/systemd/network/10-eth0.network.original
5. Edit the file with following command.
   vi /etc/systemd/network/10-eth0.network
6. Add the below lines with MTU determined in step 3, save and exit with :wq!.
   Note: MTUBytes defined as 1450 is just an example.
   [Link]
MTUBytes=1450
   
7. Restart network service:
   systemctl restart systemd-networkd
8. Try re-adding the hosts to the vCenter.