NSX Bare Metal Edge physical port fp-ethX link down event with mlx5_core driver

Products

• VMware NSX

Issue/Introduction

An NSX Bare Metal Edge node reports a physical network interface (e.g., fp-eth2) status as DOWN.

Log lines similar to the below are encountered on the Bare Metal Edge node in /var/log/syslog
mlx5_core ####:##:##.# eth4-mlx: Link down

where ####:##:##:# is the physical PCI bus address of the network card in the system. 

Environment

• VMware NSX-T Data Center
• VMware NSX 4.x
• NSX Edge Form Factor: Bare Metal

Cause

The issue is typically caused by a physical layer event detected by the NIC driver. This can be due to:

• A faulty or loose SFP module.
• A defective fiber or copper network cable.
• A physical port issue on the connected Top-of-Rack (ToR) switch.
• Intermittent signal loss resulting in the driver transitioning the interface to a down state to protect datapath integrity.

Resolution

This is not necessarily a sign of an NSX edge issue  and to rule out hardware and firmware issues, the following can be performed

Verify Physical Connectivity:

• Inspect the physical cable for damage.
• Reseat the SFP module and the cable at both the Edge node and the switch ends.
• If the issue persists, swap the SFP module or cable with a known working one.~
  
  

Check Hardware Compatibility:

• Confirm the NIC firmware version. The physical NIC firmware version can be located in system/dmesg.log on the Bare Metal Edge node (using the physical PCI bus address observed previously) 
  pci ####:##:##.#: [15b3:1017] type 00 class 0x020000
mlx5_core ####:##:#.#: firmware version: 16.28.4000
• Ensure the firmware version is compatible per the NSX Edge Bare Metal System Requirements
• If the above is all successful, the analyze the Switch Logs for any signs of why the switch port is going to a link down state.

If after validating the above and you  are contacting Broadcom support about this issue, provide the following:
Edge Log bundle
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support:
Collect Support Bundles for Troubleshooting VMware NSX
Uploading files to cases on the Broadcom Support Portal
Creating and managing Broadcom support cases