NSX-T gui system node container cluster antrea status showing down

Products

• VMware NSX
• VMware Container Networking with Antrea

Issue/Introduction

• Antrea clusters status showing down in NSX UI 
•  “Internetworking pod” is showing in “crash loopback” status however pod networking works fine. 

Environment

VMware NSX
VMware Container Networking with Antrea

Cause

Interworking Pod connects to NSX 443 (HTTPS), 1234 (NSX-RPC), 1235 (NSX-RPC). If any host firewall is configured between Antrea pods and NSX, you need to allow egress traffic to these TCP ports. 

Resolution

The source IP of these connections is interworking Pod but the Pod can be scheduled to any K8s node in the cluster. The interworking Pod uses hostNetwork (= node network) So the source IP can possible be any node IP. Hence you need to make sure we have reachability to each K8s node.  

 

 

Additional Information

VMware Ports and Protocols

1.  Interworking pods logs and tn-proxy logs would help to understand If its related to network issue.
2. Need to make sure If Interworking version is compatible with Antrea and NSX -T.
   
   Recommended compatible versions : 
   
   nsx 3.2.2  <-> antrea 1.2.3 <-> interworking 0.2
   nsx 4.0.0  <-> antrea 1.5.3 <-> interworking 0.5
   nsx 4.1.0  <-> antrea 1.7.2 <-> interworking 0.7
   nsx 4.1.1  <-> antrea 1.9.0 <-> interworking 0.9 
   
   
   
3. Get the Adapters and NSX Proxy Logs
   
   Get the name of the Antrea-interworking Pod.
   
   kubectl get pods -o wide -n vmware-system-antrea

Then note down the interworking-xxxx Pod name.
   Get the logs:
   
   kubectl logs [--tail=100] [-p] [-f] interworking-xxxx -c ccp-adapter -n vmware-system-antrea > ccp-adapter.log
kubectl logs [--tail=100] [-p] [-f] interworking-xxxx -c mp-adapter -n vmware-system-antrea > mp-adapter.log
kubectl logs [--tail=100] [-p] [-f] interworking-xxxx -c tn-proxy -n vmware-system-antrea > tn-proxy.log
   
   
4. Check if NSX Proxy Connected to APH;
   
   kubectl exec -it interworking-xxx -c tn-proxy -n vmware-system-antrea -- nsx-appctl -t /var/run/vmware/nsx-proxy/nsx-proxy-cli get/aph-conn-status
   
   Example output:
   
   {"aph connection status":[{"UUID":"########-####-####-####-########e846","STATUS":"CONNECTED"},{"UUID":"########-####-####-####-########2e12","STATUS":"CONNECTED"},{"UUID":"########-####-####-####-########6691","STATUS":"CONNECTED"}]}