NSX-T LM (Local Manager) -LM (Local Manager) communication sync time-out

Products

• VMware NSX
• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Symptoms:

• Inter-site traffic hitting default rule rather than more specific rule. ( Traffic from VM from site-1 hitting VM on site-2 hitting default-rule)
• A Group defined by dynamic criteria does NOT synchronize the dynamically discovered IP's to other local managers but you notice static IP's, in the same Group, do synchronize successfully.
  • Per the manual regarding Dynamic Groups , "Each Local Manager syncs its dynamic group membership with the other Local Managers."  .

Environment

VMware NSX-T Data Center

Cause

• LM - LM sync happens over port 1236 broken
• Remote site data entries are kept on hold for 24 hours ( default) after which would be cleaned if remote-site is disconnected.
• All data received from disconnected /delete remote site will be cleaned up after this time. ( default 24 hours)

Resolution

Always check the LM -LM sync status ( state should be in 'synched' state)

# get site-replicator remote-sites

 

Check the status of the port# 1236 on LM ( should be in established state with both GM and LM)

# netstat -nap | grep 1236

 

# nc -v lm-manager-ip 1236

 

Connections in SYN-SENT means a SYN was sent but a SYN-ACK was never received.   This is commonly caused by blocking TCP port 1236 between local managers in one site from local managers in a different sites.

Check to see if the destination receives the SYN packet by using pcap-uw.  

• Use net-stats to get the port ID ######## 

net-stats -l | grep nsx-mgr-name
########           5       9 vSwitch2         00:50:56:##:##:##  nsx-mgr-name

• Run this pcap and look for a SYN packet from the source IP.

pktcap-uw --switchport ######### --capture vNicx,VnicRx --srcip ##.##.##.## --dstport 1236 -o - | tcpdump-uw -enr -

Additional Information

Impact/Risks:

If the LM-LM syn is failing/ sync fails, DATA entries learned would be removed leading to inter-site traffic disruption after 24 hours.