ESXi hosts disconnect and storage is lost after BIOS upgrade on Dell PowerEdge Servers

Products

• VMware vSphere ESXi

Issue/Introduction

• After performing a BIOS upgrade on the physical host, multiple ESXi hosts disconnect from vCenter Server.
• Storage volumes (datastores) are no longer visible to the ESXi hosts.
• Management and storage network connectivity is lost on the original vSphere Distributed Switch (VDS).
• The original VDS was configured to use Link Layer Discovery Protocol (LLDP).
• Creating a new VDS with default settings (using Cisco Discovery Protocol (CDP)) restores connectivity for the hosts.

Cause

This issue occurs because the BIOS upgrade re-enables the embedded hardware LLDP agent within the network interface cards (NICs). 

Resolution

To resolve this issue and restore functionality to the original LLDP-enabled VDS, disable the hardware-level LLDP engine in the NIC BIOS. Contact your hardware vendor to obtain the specific firmware commands or tools required to disable the LLDP agent.

Workaround

• If the host is unreachable after upgrading the BIOS, please follow the steps to recover ESXi host connectivity when management is on DVS
  
• If your physical network environment supports it, changing the Discovery Protocol on the VDS properties from LLDP to CDP will bypass the hardware LLDP engine conflict and restore connectivity.
  • To change the Discovery Protocol o from LLDP to CDP, navigate to the Networking view in vCenter, select the VDS, and edit its settings under Manage > Settings > Properties.
  • Within the Advanced section, change the "Discovery protocol" dropdown to CDP (Cisco Discovery Protocol) and select the appropriate operation mode.

Additional Information

Risk and impact of enabling LLDP or CDP
Hosts May Disconnect from vCenter or may not forward traffic when using a X710 or XL710 or X722 family card when LLDP is enabled.
LLDP information for the Broadcom NetXtreme E-Series vmnic adapters flaps between missing and repopulating
LLDP information is visible on ESXi host but not on physical TOR switch with Mellanox adapters