NSX Bare Metal Edge Management Plane Down After Upgrading to NSX 9.1.0 when Intel E810C NIC is used for in-band management

Products

• VMware NSX

Issue/Introduction

• Following an upgrade to NSX version 9.1.0, NSX Bare Metal Edge nodes may stop functioning. The edge transport node loses connectivity, and the driver version is incorrectly updated, impacting the overlay network.
• The NSX user interface displays the error: The connection status of the edge transport node [#####-#####-####-####-###] management plane is DOWN.
• Post-upgrade, the Intel driver version is updated from ice_comms-1.3.24.0 to ice_comms-1.3.46.0.
• This issue is only seen for Intel E810-C based environment
• The issue will be seen with OOB management deployment and it will impact datapath because BFD tunnels will go down.
• The issue will also be seen with in-band Management interface deployment where Mgmt interface connectivity will be impacted along with BFD tunnels going down eventually affecting dataplane.

Environment

VMware NSX 9.1.0

Cause

This is a known issue impacting Intel E810C NICs on Bare Metal Edge nodes. During the upgrade to NSX 9.1.0, tunnels fail to establish due to incorrect VLAN handling by the updated driver. 

This issue occurs due to a firmware incompatibility between the NSX Edge DPDK driver and newer Intel E810 (ice) firmware.

When using in-band management on an E810C NIC, this DPDK incompatibility breaks management connectivity to the NSX Manager (MP/UA) and disrupts datapath VLAN traffic. Stopping the NSX dataplane service temporarily restores management access because the Edge falls back to using the native Linux driver, which is compatible with the newer firmware.

Note: Edges utilizing out-of-band management will not lose management connectivity (as they rely on the Linux driver for management traffic), but will still experience datapath disruptions due to the DPDK driver failure.

Resolution

This is a known issue impacting VMware NSX. Subscribe to this KB article to be updated if a fix version becomes available. 

If an Edge is already in the broken state, the attached script will copy the Intel ice driver from another Bare Metal Edge in the environment and patch the impacted Edge.

Requirments:

This procedure requires the presence of another working Bare Metal Edge that has not yet been upgraded
SCP/ssh as root user needs to be possible between the 2 Edges

1) Use remote or physical console access to connect to the impacted Edge as admin user and stop the dataplane service:

     stop service dataplane 

   This restores network connectivity to the management interface

2) Ensure root access is enabled.  Enable ssh root access for NSX appliances

3) SCP the attached script, deploy_ice_comms_v2.py, to /tmp on the impacted Edge.

4) Run the script and accept the related prompts:

   python3 /tmp/deploy_ice_comms_v2.py