NSX Load Balancer Server Pools Show Degraded Health for Kubernetes Services with externalTrafficPolicy Set to Local

Products

• VMware NSX
• VMware vSphere Kubernetes Service

Issue/Introduction

• The NSX Load Balancer reports a Degraded health status for server pools associated with Kubernetes Services and VKS of type LoadBalancer.
• You have persistent alarms in NSX because of this.
• This behavior only occurs for services where externalTrafficPolicy is set to Local.
• Services using externalTrafficPolicy:Cluster report a Healthy status.
• In the NSX Manager, specific pool members (worker nodes) show as Down or Unhealthy, while others remain Healthy.
• Traffic continues to flow to the Healthy members, but persistent alarms are generated in NSX.

Environment

VMware NSX 4.x
VMware NSX 9.x
vSphere Kubernetes Service

Cause

This is expected behavior due to how Kubernetes handles the Local traffic policy. When you set externalTrafficPolicy:Local, Kubernetes preserves the client source IP and avoids an extra network hop by routing traffic only to nodes that host an active pod for that service.

To facilitate this, Kubernetes allocates a healthCheckNodePort. The kube-proxy on each worker node only responds with an HTTP 200 OK on this port if a local pod is running on that specific node. If a node does not host a pod for the service, it drops the health check connection. Because the NSX Load Balancer monitors all nodes in the pool, it marks nodes without local pods as Down, leading to the Degraded status for the entire pool.

Resolution

There is currently no configuration change within NSX or the vSphere Cloud Controller Manager (CCM) to suppress this degraded status while maintaining the Local policy. The vSphere CCM does not dynamically add or remove nodes from the NSX server pool based on pod placement.

To resolve the Degraded status, you have the following options:

• Accept the Cosmetic Status: If preserving the client source IP is a requirement, you can continue to use externalTrafficPolicy:Local. The Degraded status is cosmetic as long as at least one pool member remains Healthy. NSX will continue to route traffic to the healthy nodes.
• Switch to Cluster Policy: If you do not need to preserve the client source IP, change the service configuration:
  1. Edit the Kubernetes service: kubectl edit svc <service-name>
  2. Set externalTrafficPolicy: Cluster
  3. Save the changes. This allows all nodes to respond to health checks, but the client source IP will be replaced by the node's IP.

Broadcom Engineering are aware of this issue and are planning a fix for a future release. Subscribe to this article (reference: Subscribe to a Broadcom knowledge article by article or product) to be updated on any future product enhancements regarding dynamic pool management.