Antrea cluster application not accessible when using NodePortLocal

Products

• VMware NSX
• VMware Tanzu Kubernetes Grid

Issue/Introduction

 

K8S ingress service became inaccessible with a "Secure Connection failed" error with the following setup:

• 1. Antrea CNI is used in a K8S cluster
• 2. NodePortLocal (NPL) is enabled as part of the Antrea Agent
• 3. External Load balancer (i.e, ALB)  consumes NPL port mappings published by the Antrea Agent 

 

 

The worker node iptables output has entries below:   DNAT rule without endpoint POD IP address. 

-A ANTREA-NODE-PORT-LOCAL -p tcp -m tcp --dport <port> -j DNAT --to-destination :<port>
-A ANTREA-NODE-PORT-LOCAL -p tcp -m tcp --dport <port> -j DNAT --to-destination :<port>

 

 

Antrea agent logs reports missing POD IP address:

I0918 13:26:09.700085       1 npl_controller.go:404] IP address not set for Pod: ####

 

 

Environment

Kubernetes Cluster

Antrea CNI

Tanzu vSphere

Cause

NPL rules missed the POD IP address after the Antrea Agent starts, hence traffic gets dropped at the worker node.

Resolution

There is currently no fix yet. The workarounds can be either of the following:

• Restart the antrea-agent pod
• Recreate the endpoint Pods

 

Additional Information

Antrea NodePortLocal