High CPU on NSX Bare Metal Edge Node Causing BGP and Geneve BFD Flaps

Products

• VMware NSX

Issue/Introduction

VMware NSX Bare Metal Edge nodes experience unexpected BGP and BFD session flapping. During these flap events, observation of the Edge node datapath reveals that DPDK CPU core usage is virtually maxed out (averaging ~99.4%), despite network traffic throughput remaining practically idle at only a few hundred packets per second (PPS) per core.

Example log output demonstrating high DPDK CPU utilization with minimal PPS:

"cpu_usage": {
    "dpdk_cpu_cores": 32,
    "non_dpdk_cpu_cores": 32,
    "highest_cpu_core_usage_dpdk": 99.6,
    "avg_cpu_core_usage_dpdk": 99.44,
    "highest_cpu_core_usage_non_dpdk": 15.27,
    "avg_cpu_core_usage_non_dpdk": 8.63
}

"rx_pps": {
    "0": "730 pps",
    "1": "627 pps",
    "2": "627 pps",
    "3": "313 pps"
}

tx_pps": {
            "0": "328 pps",
            "1": "406 pps",
            "2": "303 pps",
            "3": "401 pps",

Environment

VMware NSX 
VCF NSX 9.0.x

Cause

Transient CPU spikes on the DPDK cores prevent the timely processing of critical control plane traffic. In affected versions, Because Tunnel BFD packets lacked processing prioritization in the current release. Consequently, the CPU saturation delays or drops these tunnel keepalive packets, resulting in BFD and subsequent BGP session failures.

Resolution

This issue is expected to be resolved in future releases as later versions introduces Comprehensive Tunnel BFD and Tx prioritization.
This architectural enhancement prioritizes Geneve BFD packets, ensuring that tunnel BFD sessions remain stable and insulated from any transient DPDK CPU spikes.