Stale NodePortLocal DNAT rules cause external health check failures after Pod restart

Products

• VMware Container Networking with Antrea

Issue/Introduction

When a Pod with NodePortLocal (NPL) enabled is deleted and rapidly recreated, external load balancer health checks targeting the Pod may begin to fail. This occurs because the antreaagent fails to update the ANTREA-NODE-PORT-LOCAL chain in the worker node's iptables to reflect the new Pod IP.

As a result, the NodePort continues to forward incoming traffic to the old, stale Pod IP, causing the traffic to be blackholed. In cases where the affected Pod has multiple ports configured,administrators may observe a partial update some iptables rules correctly point to the new IP, while the remaining rules are stuck pointing to the old IP.

Environment

VMware Container Networking with Antrea

Cause

The issue stems from two specific flaws in how the system manages NodePortLocal routing rules during a Pod's lifecycle

• Incomplete Cleanup for Multi-Port Pods: When a Pod is terminated, the system is responsible for removing all associated network routing rules (iptables). However, if a Pod exposes multiple ports, a logic flaw causes the cleanup process to stop after processing only the first port. This leaves stale, leftover routing rules active on the node for the remaining ports.
• Missed IP Updates During Rapid Pod Recreation: When a Pod is quickly deleted and recreated with the exact same name (a common occurrence during StatefulSet restarts), the cluster events may be processed as a single "update" rather than separate "delete" and "create" events. When this happens, the system incorrectly assumes the Pod's network details are unchanged and reuses its previously cached data. Because it fails to verify if the newly created Pod was assigned a different IP address, the routing rules are never updated to point to the new IP.

Resolution

This is a known issue. The defects have been addressed in Antrea v2.3.3 and later versions (including the v2.4.x, v2.5.x, and main branches) by introducing strict validation to explicitly delete and recreate the iptables rules whenever a Pod IP changes.

To permanently resolve this issue, upgrade to a TKGm release that bundles Antrea v2.3.3 or higher.

Workaround:
If an immediate upgrade is not possible, the issue can be temporarily mitigated by restarting the antrea-agent pod on the affected worker node. When the agent restarts, it initializes an empty cache, repopulates it from current Pod annotations, and atomically writes all rules via iptablesrestore. This completely clears out any stale entries until the next time a Pod restart triggers the bug.

kubectl delete pod -n kube-system -l component=antrea-agent --field-selectorspec.nodeName=<affected-node-name>

Additional Information

Explanation of Partial Rule Updates:
The ratio of updated vs. stale rules (e.g., 4 rules updated and 4 left stale on an 8-port pod) is determined by how many times the handleRemovePod function is called during the pod deletion cycle. Because the underlying bug only deletes exactly one entry per call, the number of successfully updated  rules corresponds directly to the number of delete events (such as DeletedFinalStateUnknown events generated by the informer watch resetting) triggered before the new Pod is fully created.