Error: "no free port found" when starting pods with Antrea NodePortLocal enabled

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When creating pods enabled with nodeportlocal.antrea.io, you may encounter a "no free port found" error which prevents the pods from starting, such as:

Error syncing Pod pod-name/###-##########-#####, requeuing. Error: failed to add rule for Pod pod-name/###-##########-#####: no free port found

Deleting some running pods may temporarily resolve the issue.

Cause

Node ports are not being automatically cleared by Antrea, causing a backlog which eventually leads to no ports available.

Resolution

For now, the workaround is to restart the antrea-agent pods in the worker nodes. This will force the port recycle to proceed the way it is designed to and clean up and free up the node ports for new assignments.

This is the official workaround to be implemented until the bug is fixed at the product level. The bug fixes are included in downstream Antrea 2.3.3 and above, and Broadcom is expecting to include the fixes in the official VKR release some time in February.

Additional Information

For more information, see NodePortLocal (NPL) Documentation.

For more information on NPL allocation behavior, see GitHub: Details on NPL Allocation.

Related Knowledge Base Articles:

For more information on NPL configuration issues that may cause similar symptoms, see Antrea is consistently changing nodePortLocal for envoy Pod.