Connectivity issue between PODs using NSX Distributed Load Balancer post Tanzu for vSphere upgrade.

Products

• VMware NSX
• VMware vSphere Kubernetes Service

Issue/Introduction

• Tanzu for vSphere upgrade was completed.
• Connectivity between PODs fails.
• NSX Distributed Load Balancer (DLB) is in use and from the NSX User Interface DBL is reporting healthy.
• Capture on source POD switchport shows correct traffic.
  
  [Host] pktcap-uw --switchport [Pod Switchport] --capture VnicTx,VnicRx -o - | tcpdump-uw -enr -
[Timestamp] ##:##:##:##:##:01 >##:##:##:##:##:02:, ethertype IPv4 (0x0800), length 74: 10.10.0.10.1234 > 20.20.20.20.53: Flags [S], seq [Seq], win 64240, options [mss 1460,sackOK,TS val [Val] ecr 0,nop,wscale 6], length 0
  
  
• ESXi host uplink capture for the same traffic shows destination IP of 20.20.20.20 is still set. DLB should translate the destination IP to the endpoint IP when leaving the host.

[Host] pktcap-uw --uplink [vmnicx] --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -enr -
[Timestamp] ##:##:##:##:##:03 >##:##:##:##:##:04, ethertype IPv4 (0x0800), length 132: 30.30.30.30.5678 > 30.30.30.31.6081: Geneve, Flags [none], vni [VNI], proto TEB (TEB), options [8 bytes]:##:##:##:##:##:01 >##:##:##:##:##:02, ethertype IPv4 (0x0800), length 74:10.10.0.10.1234 > 20.20.20.20.53: Flags [S], seq [Seq], win 64240, options [mss 1460,sackOK,TS val [val] ecr 0,nop,wscale 6], length 0

• ESX host contains a stale nsxt-rtm VIB.

[ESX]$ esxcli software vib list | grep -i rtm
nsxt-rtm              0     152

Environment

VMware NSX-T Data Center 3.2.x
VMware NSX 4.x
VMware Tanzu for vSphere 

Resolution

This issue is resolved in VMware NSX 4.2.3, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround:

To workaround this issue, reboot all hosts that contain the stale RTM VIB.