Bare metal Edge node cannot claim Mellanox Technologies MT2894 Family [ConnectX-6 Lx] adapters for dataplane services

Products

• VMware NSX

Issue/Introduction

• There are multiple physical interfaces configured on the Bare Metal Edge nodes.
• In this case there were 6 interfaces configured on the node and all the interfaces are Mellanox cards.

eth# => ##.##.0 Ethernet controller [0200]: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
eth# => ##.##.1 Ethernet controller [0200]: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
eth# => ##.##.0 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
eth# => ##.##.1 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
eth# => ##.##.0 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
eth# => ##.##.1 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]

• They were configured as:
  • First 2 interfaces claimed by data-plane for overlay traffic, 
  • 4th and 6th interfaces used for out-of-band management via bond0
  • The 3rd and 5th interfaces are to be used for VLAN traffic
• Data-plane claimed the first 2 interfaces and configured fp-eth0/fp-eth1 fast-path interfaces respectively.
• However, the node is not reporting additional fp-ethX interfaces specifically fp-eth2/fp-eth3 interfaces for the remaining interfaces 3rd and 5th interfaces.
• Running the get interfaces command from NSX CLI also reports all the interfaces including the eth2, eth4 interfaces as well as the fp-eth0/1 interfaces.
• Below config.json file shows only 2 adapters for edge devicelist.

config/vmware/edge/config.json: 

"devicelist": [
        {
            "dev_type": "PCI_DEV",
            "device_id": "0x####",
            "local_cpulist": "8,9,10,11,12,13,14,15",
            "mac": "aa:bb:cc:cc:bb:aa",
            "name": "fp-eth0",
            "numa_node": "1",
            "pci": "####:##.##.0",
            "vendor": "0x####"
        },
        {
            "dev_type": "PCI_DEV",
            "device_id": "0x####",
            "local_cpulist": "8,9,10,11,12,13,14,15",
            "mac": "aa:bb:cc:cc:bb:ab",
            "name": "fp-eth1",
            "numa_node": "1",
            "pci": "####:##.##.1",
            "vendor": "0x####"
        }

• No entries are present in the syslog file regarding the creation of fp-eth2/fp-eth3 interfaces. 

YYYY-MM-DDTHH:MM:SS.184Z <Edge> DEBUG - - - eth# businfo ####:##.##.0 mac aa:bb:cc:cc:bb:aa
YYYY-MM-DDTHH:MM:SS.182Z <Edge> DEBUG - - - eth# businfo ####:##.##.1 mac aa:bb:cc:cc:bb:ab
YYYY-MM-DDTHH:MM:SS.304Z <Edge> DEBUG - - - eth# businfo ####:##.##.0 mac aa:bb:cc:cc:dd:aa
YYYY-MM-DDTHH:MM:SS.356Z <Edge> DEBUG - - - eth# businfo ####:##.##.1 mac aa:bb:cc:cc:dd:ab
YYYY-MM-DDTHH:MM:SS.411Z <Edge> DEBUG - - - eth# businfo ####:##.##.0 mac aa:bb:cc:dd:ee:aa
YYYY-MM-DDTHH:MM:SS.413Z <Edge> DEBUG - - - eth# businfo ####:##.##.1 mac aa:bb:cc:dd:ee:ab
....
YYYY-MM-DDTHH:MM:SS.483Z <Edge> DEBUG - - - PCI device ####:##.##.0 named fp-eth0 mac aa:bb:cc:cc:bb:aa acpi_index 9223372036854775807
YYYY-MM-DDTHH:MM:SS.515Z <Edge> DEBUG - - - PCI device ####:##.##.1 named fp-eth1 mac aa:bb:cc:cc:bb:ab acpi_index 9223372036854775807

• Manually adding the interfaces using command, "set dataplane device list 0000:21:00.0, 0000:22:00.0" fails with the error: "insufficient number of VFs: 0".

Environment

VMware NSX

Cause

The Mellanox Technologies ConnectX-6 Lx network interfaces are not currently supported for use with NSX Bare Metal Edge nodes. As a result, while these interfaces can still be utilized for Edge management traffic, they cannot be claimed or used by the NSX data plane to provide networking services.

Resolution

It is recommended to configure the Bare Metal Edge node with network adapters that are supported to ensure all necessary interfaces can be used by the NSX data plane.

Additional Information

For more information regarding NSX Edge Bare metal Requirements, please refer: NSX Edge Bare Metal Requirements
And for details regarding NSX Bare metal edge node installation, please refer: Install Bare Metal NSX Edge Automatically using ISO File