LoadBalancer Service gets recreated in vSphere Kubernetes Guest Cluster after deleting the LB Service
search cancel

LoadBalancer Service gets recreated in vSphere Kubernetes Guest Cluster after deleting the LB Service

book

Article ID: 421979

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware vSphere ESXi

Issue/Introduction

  • Stale LoadBalancer service within a vSphere Kubernetes cluster gets recreated after it is attempted to be deleted with the following command:
    kubectl delete svc <svc_name>
  • The services are identified as excessive or stale and are running on a specific port, such as port 9094.

Environment

VMware vSphere Kubernetes Service

Cause

The continuous recreation of the LoadBalancer Service is caused by Kubernetes Reconciliation, specifically triggered by the VM Operator component. The LoadBalancer Service is not a standalone object, it is a child resource managed by a parent Custom Resource called VirtualMachineService.

The presence of the ownerReferences field in the LoadBalancer Service's YAML contains this relationship:

ownerReferences:
  - apiVersion: vmoperator.vmware.com/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachineService
    name: <name of the VirtualMachineService>
    uid: <UID of VirtualMachineService>


The VM Operator, which manages the lifecycle of VMs and associated resources, constantly watches the parent VirtualMachineService object. When the child LoadBalancer Service is deleted, the operator detects a drift from the desired state and automatically recreates the missing LoadBalancer Service to restore the configuration.

Resolution

To permanently delete the LoadBalancer Service, the parent VirtualMachineService custom resource should be deleted. This action triggers Kubernetes Garbage Collection to automatically and safely delete the dependent child service.

  1. Identify the Parent Object Name: Retrieve the YAML of the affected LoadBalancer Service and note the exact name of the VirtualMachineService from the ownerReferences section (which you are trying to delete).

  2. Delete the Parent VirtualMachineService: Execute the following kubectl command, ensuring the correct name and namespace is been used. This command deletes the parent object and trigger the cascade deletion of the child LoadBalancer Service:

    kubectl delete virtualmachineservices <Name of the VirtualMachineService> -n <Namespace Name>
    
  3. Validate Deletion: Run kubectl get svc -n <namespace_name> to confirm that the LoadBalancer Service is no longer listed and is not being recreated. Repeat the process for all excessive services within the namespace.

Note: The VirtualMachineService resource is designed to expose VM workloads. Deleting this object confirms that the underlying VM workload no longer needs external load balancing.