VKS Workload Node Pool Intermittently NotReady: Packet Loss and Gateway Unreachable

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

A VMware vSphere Kubernetes Service workload cluster node pool experiences status instability, where nodes repeatedly toggle between Ready and NotReady states. MachineHealthCheck controllers continuously delete and re-provision affected nodes in an auto-healing loop. Diagnostic checks reveal 100% packet loss to the default gateway, API server timeouts, and stale/delayed ARP entries in the kernel neighbor table for specific node IPs.

Symptoms & Diagnostic Evidence

1. 100% Gateway Packet Loss & Control Plane Isolation Executing network diagnostics from an affected node shows 100% ICMP packet loss to the default gateway and an inability to establish TCP connections with the Kubernetes API server:

Bash

 

node-user@vks-node-pool-np1-xxxxx:~$ ping -c 5 <DEFAULT_GATEWAY_IP>
PING <DEFAULT_GATEWAY_IP> 56(84) bytes of data.
--- <DEFAULT_GATEWAY_IP> ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4096ms

2. Gateway Neighbor Entry Stuck in DELAY / STALE State Inspecting the Linux neighbor table on eth0 shows healthy node-to-node connectivity, but the default gateway IP gets stuck in DELAY or STALE states due to ARP resolution failures:

Bash

 

node-user@vks-node-pool-np1-xxxxx:~$ ip neighbour show dev eth0
172.23.10.135 lladdr 00:50:56:xx:xx:xx REACHABLE
172.23.10.129 lladdr 08:c0:eb:xx:xx:xx DELAY
172.23.10.130 lladdr 08:c0:eb:xx:xx:xx STALE

3. External Ping Discrepancy External pings to newly provisioned worker node IPs fail intermittently or time out, while pings to other nodes on the same subnet succeed with 0% packet loss.

Environment

 VMware vSphere Kubernetes Service 

Cause

An IP address conflict (duplicity) exists between the VKS Supervisor Namespace IP Pool and the NSX Advanced Load Balancer (Avi) Service Engine (SE) IP allocation pool.

When Avi Service Engines and VKS worker nodes draw IP addresses from overlapping ranges on the same port group/VLAN, both virtual interfaces attempt to claim the same IP. The resulting IP conflict triggers Layer 3 routing failures and Dynamic ARP Inspection / IP Source Guard drops on the physical gateway, causing kubelet heartbeats to fail.

Resolution

Step 1: Audit and Segregate Network IP Pools

Ensure strict separation between the Kubernetes Node IP ranges and all external network allocation pools sharing the port group:

1. Review IPAM and Network Management sheets for the assigned VLAN.

2. Identify all active allocation ranges (e.g., Load Balancer VIP/SE pools, DHCP scopes, static device ranges).

3. Adjust external service pools or DHCP server exclusions so their ranges do not overlap with the CIDR block designated for VKS Worker Nodes.

Step 2: Adjust Supervisor Namespace IP Pool in vCenter (If Required)

If external static ranges cannot be changed, update the VKS Supervisor Namespace IP Pool bounds to start at an unassigned, dedicated IP range:

1. Open vCenter -> Workload Management -> Namespaces -> Select your Namespace.

2. Edit the IP Pool settings under the Network configuration.

3. Shift the pool range to begin at a verified, non-conflicting IP address block.

Step 3: Remove Conflicted Nodes (https://knowledge.broadcom.com/external/article/391365/delete-specific-worker-node-when-scale-i.html)

To safely remove the conflicted machine object and force Cluster API to provision a clean VM:

1. Apply the deletion annotation to the target machine object in the Supervisor Namespace:

   Bash

    

   kubectl annotate machine <machine-name> -n <supervisor-namespace> "cluster.x-k8s.io/delete-machine"="yes"
   

2. Delete the machine object:

   Bash

    

   kubectl delete machine <machine-name> -n <supervisor-namespace>
   

Step 4: Validate Cluster Health

Verify that newly provisioned worker nodes obtain non-conflicting IP addresses, establish REACHABLE gateway states, and remain permanently in Ready status:

Bash

 

kubectl get nodes -o wide -w