NSX Management Proxy is not compatible with VC 8.0 + NSX Advanced LB (Avi)

Products

• VMware Avi Load Balancer
• VMware NSX
• VMware NSX-T Data Center
• VMware vCenter Server 8.0
• VMware Container Networking with Antrea

Issue/Introduction

Deployed NSX Management Proxy Supervisor Service and the deployment was successful. However, the communications to ports 10082 and 10081 are down.

• Below image is from Avi Web UI→Virtual Services→search for nsx-management-proxy.

• antrea-interworking deployment in workload cluster is down.

This issue causes the Antrea-NSX integration to fail. Affected Versions:

• nsx-management-proxy v0.1.1, v0.2.0 and v0.2.1

Environment

VMware Avi Load Balancer

VMware NSX

VMware NSX-T Data Center

VMware vCenter Server 8.0

VMware Container Networking with Antrea

Cause

• The issue is caused by nsx-management-proxy Pod and the K8s LB Service proxy-loadbalancer being realized under different Tier-1 gateways. Therefore, the Avi SE cannot establish connection with the nsx-management-proxy Pod. 

• Avi SE is attached to a segment under the Tier-1. Therefore, Avi SE cannot use the Tier-1 uplink IP for its traffic, instead, its using the segment subnet. There is also route advertisement rule for Tier-1 to deny advertising the Avi SE segment to other Tier-1s.

• Avi SE(under nsx-management-proxy Tier-1) can send a TCP SYN packet to the nsx-management-proxy Pod on another Tier-1(Supervisor Control Plane VM's Tier-1), but the response TCP SYN+ACK is dropped at the Supervisor Control Plane VM's Tier-1, because Supervisor Control Plane VM's Tier-1 doesn't know how to send the packet back to Avi SE (which is under nsx-management-proxy Tier-1).

Resolution

Workaround:

• The workaround is adding an SNAT rule to the nsx-management-proxy  Tier-1 for the Avi SE egress traffic.
  Detailed steps are mentioned below.
• Find out the T1 uplink IP and Avi segment subnet CIDR under the nsx-management-proxy  Tier-1.
  • In NSX UI, click Networking -> Tier-1 Gateways -> filter nsx-management-proxy  in the listing -> expand the Tier-1 gateway configuration -> Additional Settings -> Router Links
    Note down the router link IP

• Copy avi segment name from above Linked Segments and use it to filter under Networking→Segments and note down avi segment subnet CIDR.

• Add an SNAT rule to the nsx-management-proxy Tier-1.
  • In NSX UI, click Networking → NAT → select gateway → Add NAT RULE
  • Source = Avi segment subnet CIDR, Translated IP = T1 uplink IP.

• An example SNAT rule after addition.