The assigned LoadBalancer ingress IP does not match the corresponding VirtualMachineService IP on the Supervisor layer in VKS environment with HAproxy

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

 

The configuration mismatch was verified by comparing resource states across both cluster layers.

For example, in the guest cluster namespace, the service reported a LoadBalancer Ingress IP of "10.XXX.XXX.XX1". However, checking the matching VirtualMachineService resource on the Supervisor layer inside namespace showed a conflicting status Ingress IP of "10.XXX.XXX.XX2". You can observe the issue by running below commands: 

# in the guest cluster:

$ kubectl describe svc -n namespace        svcName

# in the supervisor cluster:

$ kubectl describe virtualmachineservice -n namespace  virtualmachineserviceName

 

This issue was observed by recreating the service from application side with an identical name after deletion.

Environment

vSphere Kubernetes Service with HAProxy

Cause

The IP drift is caused by a race condition resulting from propagation delays between the guest cluster cloud provider and the Supervisor layer. When a virtual service is deleted, the Supervisor and underlying load-balancing infrastructure (e.g., HAProxy) process the de-provisioning asynchronously. Recreating the service with an identical name immediately after deletion causes the guest cluster to reference stale or unreleased virtual IP (VIP) allocations before the Supervisor layer has finalized the cleanup.

It takes up to around 5-10 minutes for the Supervisor/HAProxy layer to completely release a deleted VIP. Recreating the same service within this window will cause a race condition/mismatch.

Isolating the issue showed that creating entirely new services faced no mismatch, whereas immediate re-use of an identical service name consistently failed. Testing verified that allowing a 15-minute buffer allowed the asynchronous infrastructure state to settle, which proves the mismatch stems from timing and infrastructure sync lag rather than a permanent component failure.

 

Resolution

 

• Apply Operational Delay: Ensure a time interval of at least 15 minutes is maintained between the deletion and recreation of a virtual service if reusing the identical service name.

• Optimize Deployment Pipelines: Modify application deployment automation workflows to utilize unique or version-appended service names to prevent immediate identifier re-use.

• Clear Cached Network State: In the event of an active mismatch, clear the drifted routing definitions by restarting the kube-proxy, antrea and vmware-cloud-provider pods on the affected Guest Cluster.

 

Additional Information

Please note the HAProxy - Supportability -

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/vsphere-supervisor-concepts-and-planning/vsphere-with-tanzu-deployment-options/topologies-for-deploying-the-haproxy-load-balancer.html

"A support contract is required with HAProxy to get technical support for the HAProxy load balancer. VMware GSS cannot provide support for the HAProxy appliance."