NSX Native Load Balancer fails to load on Bare-Metal Edge

Products

• VMware NSX

Issue/Introduction

In NSX 4.x environments, the native NSX Load Balancer (LB) may fail to function when deployed on Bare-Metal Edge (BME) nodes. The same configuration works when the Tier-1 gateway is hosted on Virtual Edge nodes, but fails when moved to BMEs.

Customers may observe that the LB configuration appears valid, but traffic is not processed by the load balancer.

Symptoms

• Load Balancer VIP is unreachable
• Clients receive TCP RST
• Traceflow shows traffic being forwarded instead of terminated at VIP
• LB diagnostic output shows:
  • not_found_in_db: LbCurrentMsg
  • cfg: ha is not ready

Environment

VMware NSX 4.x 
Bare-Metal Edge Nodes
Native NSX Load Balancer

Cause

When service_corelist is configured on a Bare-Metal Edge, NSX reserves CPU cores for service-core processing (non-datapath services).

This setting is defined via the custom block on the BME config json:

• /config/vmware/edge/config.json
  • eg: "service_corelist": "1,5,9,7,3,13,17,19,15,11"

This configuration automatically disables the native NSX Load Balancer on BME nodes.

This is by design and has existed since earlier NSX versions.

Although the LB configuration is successfully applied at the policy layer:

• The LB service is not instantiated at runtime
• The LB container/process is terminated or never started
• The LB runtime state is not created

Because the LB service is disabled at runtime:

• LB configuration is accepted at the policy layer
• Missing runtime object (LbCurrentMsg)
• LB HA never becoming ready
• vIP never binds

When service_corelist is populated on a Bare-Metal Edge, NSX reserves the specified CPU cores for service-core threads:

These cores are used for:

• L7 Security Pipeline
• DPI / Context-Aware Firewall (CAFW)
• L7 Application Identification (App-ID)
• HyperScan
• IDPS

Resolution

The service cores on the BME could be disabled, but the following features sets would be lost:

• L7 security pipeline
• DPI/CAFW
• L7AP/app-ID
• HyperScan
• IDPS

Configuration of service cores is documented here:

• https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/installing-nsx-edge/manually-deploying-nsx-edge-node-outside-of-nsx/edit-nsx-edge-transport-node-configuration.html

Additional Information

• This limitation applies only to Bare-Metal Edge nodes.
• Virtual Edge nodes are not impacted
• Behavior is consistent across NSX versions where service_corelist is enabled on BME