Supervisor enablement stuck on Foundation Load Balancer (FLB) deployment. Error "Timed out waiting for LB service update"

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

When deploying supervisor with Foundation Load Balancer (FLB) could get stuck at FLB deployment stage.

The following symptoms are observed:

• Supervisor control plane VMs were deployed successfully but the FLB VMs are not present in the inventory.
  
  
• The Supervisor Management page shows the Supervisor cluster in Configuring state and will time-out with the error:
  

  Configured Load Balancer fronting the kubernetes API Server
  Timed out waiting for LB service update. This operation is part of the cluster enablement and will be retried

  


• On the vCenter VM, the following logs indicate Solution task failures for FLB and reference the domain ID for the deploying Supervisor cluster:
  
  
  • /var/log/vmware/vpxd/vpxd.log
    

    YYYY-MM-DDTHH:MM:SS error vpxd[#####] [Originator@1234 sub=FoundationLoadBalancersSvc opID=FLB-Monitor-CheckCompliance-######] vLCM API invocation failed; cluster: domain-c####, method: SetSolution, taskId: ######-####-####-####-#######:com.vmware.esx.settings.clusters.vms.solutions, error: Task failed with Unknown Error: Unknown Exception
    YYYY-MM-DDTHH:MM:SS error vpxd[#####] [Originator@1234 sub=FoundationLoadBalancersSvc opID=FLB-Monitor-CheckCompliance-######] Failed to set solution; cluster id: domain-c####, solution id: foundation-loadbalancer-1,

  • /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

    YYYY-MM-DDTHH:MM:SS info vmware-vum-server[11070] [Originator@1234 sub=com.vmware.vcIntegrity.lifecycle.lccm.SetSolutionTask] [SolutionsSvcSetTask, 71] Task:com.vmware.vcIntegrity.lifecycle.lccm.SetSolutionTask ID:######-####-####-####-#######. Storing solution "foundation-loadbalancer-1" for cluster "domain-c####"
    YYYY-MM-DDTHH:MM:SS error vmware-vum-server[11070] [Originator@1234 sub=Default] [VdbStatement] Execute result code: -1
    YYYY-MM-DDTHH:MM:SS error vmware-vum-server[11070] [Originator@1234 sub=Default] [VdbStatement] SQL execution failed: INSERT INTO SYSTEM_VMS_CLUSTER_SOLUTION_SPEC_VM_PLACEMENT_POLICIES (CLUSTER_ID, SOLUTION_ID, VM_PLACEMENT_POLICY) VALUES (?, ?, ?)
    YYYY-MM-DDTHH:MM:SS error vmware-vum-server[11070] [Originator@1234 sub=Default] [VdbStatement] Execution elapsed time: 1 ms
    YYYY-MM-DDTHH:MM:SS error vmware-vum-server[11070] [Originator@1234 sub=Default] [VdbStatement] Statement diagnostic data from driver is 42703:0:1:ERROR: column "vm_placement_policy" of relation "system_vms_cluster_solution_spec_vm_placement_policies" does not exist;
    --> Error while preparing parameters

Environment

vCenter Server 9.x

Cause

The execution of the vum-db-reset (/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db) operation on vCenter Server removed essential database entries required for lifecycle operations.

Resolution

Resolve inconsistencies with the VMware Update Manager (VUM) database by following the below KB article:

Fixing VUMDB inconsistency issues after running vLCM reset script on 8.0U3 to 9.0.x

Additional Information

Reset VMware Lifecycle Manager depot in vCenter Server Appliance

Creating and managing Broadcom support cases