Inter-node Pod network communication failure VMware Tanzu Kubernetes Grid

Products

• VMware Container Networking with Antrea
• VMware Tanzu Kubernetes Grid

Issue/Introduction

In VMware Tanzu Kubernetes Grid (TKG) 2.5.4 environments using Antrea CNI, applications such as Harbor may experience intermittent network communication failures between Pods located on different nodes. The issue is specifically characterized by

• Intermittent connection drops (e.g., from harbor-core to harbor-database).
• Traffic entering the destination node via the Antrea tunnel (antrea-tun0) but failing to reach the target Pod.
• The following error may be observed in kube-proxy logs "Failed to retrieve node IPs" err="host IP unknown; known addresses: []"
• A tcpdump on the destination node shows incoming SYN packets, but a tcpdump on the Pod's internal interface shows no incoming traffic.

Environment

VMware Container Networking with Antrea

VMware Tanzu Kubernetes Grid (VMware Tanzu Kubernetes Grid)

Cause

The issue is caused by a race condition resulting in duplicate Pod IP assignments on a single node. When short-lived ephemeral Pods are assigned an IP address already in use by a long-running Pod, a conflict occurs in the Open vSwitch (OVS) forwarding pipeline.

Specifically, the Antrea L3Forwarding table (Table 72) matches rules strictly based on the destination IP. When the short-lived Pod is deleted, the antrea-agent removes the associated OpenFlow entry for that IP. Because the IP is shared, the removal process also strips the routing logic for the original, long-running Pod, causing all subsequent traffic to that IP to be dropped.

Resolution

To restore connectivity and address the underlying state drift, follow these steps

1. Restart the antrea-agent and kube-proxy Pods on the affected worker node. This forces the agent to re-synchronize the OVS pipeline with the Kubernetes API server and restores the missing L3Forwarding flows.

   kubectl delete pod -n kube-system <antrea-agent-pod-name>kubectl delete pod -n kube-system <kube-proxy-pod-name>

2. Verify that the L3Forwarding flow has been successfully reprogrammed by running the following command against the restarted antrea-agent

   kubectl exec -it -n kube-system <antrea-agent-pod-name> -c antrea-agent -- ovs-ofctl dump-flows br-int -O OpenFlow13 table=L3Forwarding | grep '<affected_pod_ip>'

   Note: Replace <affected_pod_ip> with the IP of the Pod that was experiencing drops.

3. The duplicate assignment typically occurs when the host-local IPAM cache becomes out of sync.

   • Audit the node for any automated scripts, security agents, or systemd timers that may be performing filesystem cleanups.
   • Ensure that no processes are modifying or deleting files within the CNI network directory: /var/lib/cni/networks/antrea/.