BME Tunnel Failure Caused by NIC PCI ID Change After Firmware/BIOS Upgrade

Products

• VMware NSX

Issue/Introduction

Bare Metal Edge (BME) tunnel connectivity was lost after the NIC firmware and BIOS upgrade. From the /edge/docker_logs_service_datapath log, it shows that the fp-ethx interface does not exist, where x corresponds to the fast‑path interface number. An example log snippet is shown below.

docker_logs_service_datapath:YYYY-DD-MMTHH:MM:SSZ datapathd 7393 dpc-pb tname="dp-ipc72" [ERROR] Iface for port name fp-ethx does not exist errorCode="EDG0400104"
[...]
docker_logs_service_nsxa:YYYY-DD-MMTHH:MM:SSZ nsxa 1 dp-ipc [ERROR] Receive error ack version 1 from DP: Iface for port name fp-ethx does not exist errorCode="EDG0100372"

From the /var/log/kern.log, the PCI device ID for ethx changed from 0000:#d:00.1 (ethx / enp173s0f1). This timestamp aligns with when NSX BME began reporting that fp‑ethx no longer existed, as noted earlier. An example log snippet is shown below.

YYYY-DD-MMTHH:MM:SSZ <BME hostname> kernel - - - [ 18.312111] ice 0000:#d:00.1 ethx: renamed from enp##3s0f1
[...]
YYYY-DD-MMTHH:MM:SSZ <BME hostname> kernel - - - [ 18.155778] ice 0000:#e:00.1 ethx: renamed from enp##4s0f1

 

 

Environment

VMware NSX

Cause

This issue can occur if the NIC’s PCI ID changes after a firmware or BIOS upgrade. BME normally auto‑detects supported NICs but when a custom device list is configured using the "set dataplane device list" command, this overrides the automatic detection process.

This can be verify this by reviewing the BME node configuration under /config/vmware/edge/config.json and checking whether the custom_devicelist section is present. An example is shown below.

[...]

    "custom": {
        "custom_devicelist": [
           "0000:##:00.0",
           "0000:##:00.1",
           "0000:##:00.2"
       ],

[...]

Resolution

If a specific fast‑path (FP) configuration is used for custom PCI IDs, the user must verify whether any PCI IDs have changed after a firmware or BIOS upgrade. If any changes are found, the FP configuration must be updated to include the new PCI ID using the command below.

set dataplane device list <a list of PCI device IDs including the new PCI ID in comma‑separated format>

See the example below.

set dataplane device list 0000:04:00.0,0000:0b:00.0,0000:13:00.0,0000:1b:00.0

Refer to KB 413089 for additional details.