Service Engine Fails to Connect to Avi Controller Node After Host Upgrade to OEL 8.10

Products

• VMware Avi Load Balancer

Issue/Introduction

• Service Engines (SEs) container fail to connect back to the Avi Controller after the underlying host operating system is upgraded from Oracle Enterprise Linux (OEL) 7.x to OEL 8.10. This issue is observed on bare-metal deployments where the SE container runs successfully but cannot send network packets out after taking over the Network Interface Card (NIC).

Environment

 

• Avi Load Balancer Version: 22.1.6 (or similar versions running DPDK 20.05).

• Host OS: OEL 8.10 (upgraded from OEL 7.x).

• Affected NIC: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28.

• NIC Firmware: 6.01 (which is quite old and may not go well with OEL8.10 version to connecting back to controller issue)

 

Cause

The core problem is an incompatibility between the old NIC firmware version (6.01) with OEL 8.10, which conflicts with the DPDK version used by the Avi Service Engine.

• The Avi ALB version 22.1.6 SE uses DPDK version 20.05

• To check the current NIC details and firmware version on the non-working SEs, run the following commands:

  for iface in $(ls /sys/class/net/); do echo "=== $iface ==="; ethtool -i $iface; echo ""; done
  lspci | grep -i net
  lshw -c network -sanitize

• The OEL 8.10 environment, combined with the outdated NIC firmware (e.g., 6.01) for the Intel XXV710, results in a failure when the SE container's DPDK attempts to take over the NIC for data plane operations.

• Specifically, the SE is unable to send packets out after NIC takeover, preventing communication with the Controller and leading to a disconnected state. The NIC firmware version 6.01 is confirmed to be quite old and incompatible with the OEL 8.10 host environment in this context.

Resolution

The primary resolution is to upgrade the NIC firmware on the OEL 8.10 host to a version officially compatible with the host OS and the relevant DPDK version.

• Upgrade NIC Firmware: Upgrade the NIC Firmware to 7.03 (minimum) or higher version (9.01). This newer firmware is necessary to ensure compatibility with the OEL 8.10 host environment and the DPDK version utilized by the Avi Service Engine.

• Upgrade NIC Drivers:

  1. Refer the link - Kernel driver and Firmware Matching List

  2. Link to download the firmware: Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Network Adapter 700 Series

  3. Installation steps: Upgrading the Intel NIC Firmware

• Re-enable Host: After the NIC firmware upgrade, re-add the OEL 8.10 host to the Avi cloud. The SE container should now be able to take over the NIC correctly and connect back to the Controller.

Additional Information

 

• This issue is specific to the interaction between the host OS version, the NIC model, the NIC firmware, and the DPDK version used by the Avi SE.

• The version of DPDK used in Avi ALB 22.1.6 is 20.05.

• 30.2.x version is running with dpdk-22.11 version. Require NIC Firmware version on 9.01