ESXi host randomly disconnects from vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

When using Jumbo frames for the management interface for ESXi, the hosts will randomly disconnect from vCenter.  

Attempting to ping vCenter with a large packet size fails:

vmkping -I vmk0 <vcenter_IP> -d -s 8972   
Packets: Sent = 10, Received = 0, Lost = 10 (100% loss)

Using a smaller size succeeds:

vmkping -I vmk0 <vcenter_IP> -d -s 1472
Packets: Sent = 10, Received = 10, Lost = 0 (0% loss)

From the vCenter Server /var/log/vmware/vpxd/vpxd.log file you will see an error similar to the following: 

####-##-##T##:##:##.###+##:## warning vpxd[2420870] [Originator@6876 sub=HttpConnectionPool-000001 opID=HB-host-6059@924601-3f6d5859-WorkQueue-e1122ee] Failed to get pooled connection; <cs p:00007fd7d1eb3030, TCP:host-ip:443>, SSL(<io_obj p:0x00007fd710e34e90, h:414, <TCP 'vcenter-ip : 44254'>, <TCP 'host-ip : 443'>>), duration: 7msec, N7Vmacore15SystemExceptionE(stream truncated: The connection was closed by the remote end during handshake.)
--> [context]zKq7AVECAQAAAA8jcwENdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoAKHI4ANofQQCpJkEAvkFBAATsNwAXRTgAuw9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]

Environment

ESX 7.x

ESX 8.x

vCenter Server 7.x

vCenter Server 8.x

Cause

This is caused by an improper MTU configuration within the environment.  When network frame size is set to jumbo frames in an ESXi host's vmkernel NIC configuration, those frames might be dropped or fragmented by any other device that is set to only pass standard sized frames. Therefore, the frame size setting for all devices along the network path must allow jumbo frames to pass in order for the communication to work. 

Resolution

Reconfigure the management interface for ESX to use a smaller MTU or increase the MTU size for the devices upstream of the ESX host.

 

Additional Information

Enabling Jumbo Frames on virtual switches

Testing jumbo frame pings from ESXi over the network