Host disconnected/not-responding in vCenter - i40en Link State Change Errors

Products

• VMware vSphere ESXi

Issue/Introduction

It is often the case that unexpected link state changes may be reported by ESXi / vCenter for physical uplinks (vmnics) associated with one or more ESXi host(s). 
When this occurs, the objective is to determine the likely cause. It can be expected behaviour  - like this KB suggests, however there is a known, physical reason for i40en vmnic link status change issues.

 

/var/run/log/vmkernel.log
WARNING: i40en: i40en_LinkEvent:: 0000:xx:00.0: Get Link Status - EAGAIN error code received
WARNING: i40en: i40en_LinkEvent:: 0000:xx:00.0: Link is down. The installed module is corrupted or a module communication error occurred
WARNING: i40en: i40en_LinkEvent:: 0000:xx:00.0: Couldn't get link state, status: -53, link polling on
WARNING: i40en: i40en_LinkEvent:: 0000:xx:00.0: Get Link Status - EAGAIN error code receive

Environment

VMware vSphere ESXi

Cause

The logs show a lot of messaging for i40en_LinkEvents. ESXi can not get a response from the physical port/NIC. Highlighted in the log snippet above, is the physical PCI device port ID. For example : 0000:15:00.0.

The EAGAIN error tells us the driver is losing communication with the card's firmware because the hardware is failing or resetting. ESXi is unable to determine the link state of the vmnic,  and as a result loses connectivity on those affected vmnics. Which in turn can cause the host to appear disconnected, or as unresponsive in the vCenter inventory since vCenter could not determine a heartbeat response from the host in time. 

 

Resolution

Check for recent driver/firmware changes and that the drivers/firmware are up to date using the VMware by Broadcom compatibility guide  - compatibilityguide.broadcom.com. See KB - Supported drivers and firmware versions for I/O devices

This may require further guidance from your hardware vendor if the issues persists.