Error: "Failed to find any suitable datastore" during VCF 9.0 Deployment with "Timed out waiting for LB service update" with FLB Deployments
search cancel

Error: "Failed to find any suitable datastore" during VCF 9.0 Deployment with "Timed out waiting for LB service update" with FLB Deployments

book

Article ID: 446597

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

When enabling vSphere with Tanzu (WCP) or upgrading the Supervisor, the deployment of the Foundation Load Balancer (FLB) fails and becomes stuck in a "Configuring" or NON_COMPLIANT loop.

  1. Timed out waiting for LB service update while activating Supervisor with VDS is seen under the deployment stages:

  2. In /var/log/vmware/wcp/wcpsvc.log, the cluster enablement fails:

    error wcp [kubelifecycle/spherelet_lifecycle.go:839] [opID=69f5786e] Cluster version not supported for cluster domain-c5674870

  3. In /var/log/vmware/vpxd/vpxd.log, the vCenter database shows a stuck solution application:

    Skip apply solution request since another one is in progress; <ID>:com.vmware.esx.settings.clusters.vms.solutions

  4. In /var/log/vmware/vpxd/vpxd-*.log, vSphere DRS drops the primary vSAN datastore due to a size boundary limitation

    vpxd-####.log.gz:####-##-##T##:##:##.###Z warning vpxd[#####] [Originator@#### sub=sdrsLogger opID=########] Datastore [vim.Datastore:datastore-#######,<CATEGORY_REDACTED>] invalid size: capcaityBytes=########, freeBytes=########
    vpxd-####.log.gz:####-##-##T##:##:##.###Z error vpxd[#####] [Originator@#### sub=drmLogger opID=########] Received Fault while building context for place disk: (vim.fault.NoCompatibleDatastore) {
    vpxd-####.log.gz:####-##-##T##:##:##.###Z error vpxd[#####] [Originator@#### sub=drmLogger opID=########] Can not place disk.
    vpxd-####.log.gz:####-##-##T##:##:##.###Z info vpxd[#####] [Originator@#### sub=vpxLro opID=########] [VpxLRO] -- FINISH lro-########
    vpxd-####.log.gz:####-##-##T##:##:##.###Z warning vpxd[#####] [Originator@#### sub=sdrsLogger opID=########] Datastore [vim.Datastore:datastore-#######,<CATEGORY_REDACTED>] invalid size: capcaityBytes=########, freeBytes=########
    vpxd-####.log.gz:####-##-##T##:##:##.###Z error vpxd[#####] [Originator@#### sub=drmLogger opID=########] Received Fault while building context for place disk: (vim.fault.NoCompatibleDatastore) {
    vpxd-####.log.gz:####-##-##T##:##:##.###Z error vpxd[#####] [Originator@#### sub=drmLogger opID=########] Can not place disk.
    vpxd-####.log.gz:####-##-##T##:##:##.###Z info vpxd[#####] [Originator@#### sub=vpxLro opID=########] [VpxLRO] -- BEGIN lro-######## -- SessionManager -- vim.SessionManager.sessionIsActive -- ########-####-####-####-############(########-####-####-####-############)
    vpxd-####.log.gz:####-##-##T##:##:##.###Z info vpxd[#####] [Originator@#### sub=vpxLro opID=########] [VpxLRO] -- BEGIN lro-######## -- session[########-####-####-####-############]########-####-####-####-############ -- vim.view.View.destroy -- ########-####-####-####-############(########-####-####-####-############)
    vpxd-####.log.gz:####-##-##T##:##:##.###Z info vpxd[#####] [Originator@#### sub=vpxLro opID=########] [VpxLRO] -- FINISH lro-########

  5. In /var/log/vmware/eam/eam.log, ESX Agent Manager throws an exception as it fails over to an invalid local datastore:

    Download:Route(routedUrl:file:/storage/lifecycle/vmware-halb/photon-ova.ovf, url:file:/storage/lifecycle/vmware-halb/photon-ova.ovf, pemCertificate:null, thumbprint:null, headers:{}:ac3494c8cffb699b] Downloaded OVF descriptor from URL: file:/storage/lifecycle/vmware-halb/photon-ova.ovf
    ####-##-##T##:##:##.###Z | INFO | cluster-agent-1 | DeployVmJob.java | 574 | Deploying agent on cluster <CATEGORY_REDACTED> (ClusterComputeResource:domain-c5674870) with name <HOSTNAME_REDACTED> (67)
    ####-##-##T##:##:##.###Z | INFO | cluster-agent-4 | DeployVmJob.java | 574 | Deploying agent on cluster <CATEGORY_REDACTED> (ClusterComputeResource:domain-c5674870) with name <HOSTNAME_REDACTED> (68)

    ...

    ####-##-##T##:##:##.###Z | WARN | vim-async-1 | PlaceVmsXCluster.java | 103 | [RecommendDatastore] Failed to place VM: <HOSTNAME_REDACTED> (67) due to: [(vim.fault.GenericDrsFault) {
       faultCause = null,
       faultMessage = (vmodl.LocalizableMessage) [
          message = Failed to find any suitable datastore in the destination cluster.
       hostFaults = null
    ####-##-##T##:##:##.###Z | WARN | cluster-agent-1 | Retry.java | 111 | Failed to Deploying VM for LCCMAgent(cluster-bound, ID:) at attempt 1 out of 5 Reason: com.vmware.eam.EamRemoteSystemException: Failed to get placement recommendation virtual machine in the cluster
       agencyName = 'foundation-loadbalancer-6',
       agencyName = 'foundation-loadbalancer-6',
    ####-##-##T##:##:##.###Z | ERROR | cluster-agent-1 | AuditedJob.java | 106 | JOB FAILED: [#########] DeployVmJob(LCCMAgent(cluster-bound, ID: ca744bd1-b271-4ca7-a4e6-762d2dfc5375))
       entityName = <HOSTNAME_REDACTED> (68),
    ####-##-##T##:##:##.###Z | WARN | vim-async-1 | PlaceVmsXCluster.java | 103 | [RecommendDatastore:] Failed to place VM: <HOSTNAME_REDACTED> (68) due to: [(vim.fault.GenericDrsFault) {
       faultCause = null,
       faultMessage = (vmodl.LocalizableMessage) [
          message = Failed to find any suitable datastore in the destination cluster.
       hostFaults = null
    ####-##-##T##:##:##.###Z | WARN | cluster-agent-4 | Retry.java | 111 | Failed to Deploying VM for LCCMAgent(cluster-bound, ID:) at attempt 1 out of 5 Reason: com.vmware.eam.EamRemoteSystemException: Failed to get placement recommendation virtual machine in the cluster
       agencyName = 'foundation-loadbalancer-#',
       agencyName = 'foundation-loadbalancer-#',
    ####-##-##T##:##:##.###Z | ERROR | cluster-agent-4 | AuditedJob.java | 106 | JOB FAILED: [#] DeployVmJob(LCCMAgent(cluster-bound, ID: ))
    com.vmware.eam.EamRemoteSystemException: Failed to get placement recommendation virtual machine in the cluster
    ####-##-##T##:##:##.###Z | ERROR | cluster-agent-1 | AuditedJob.java | 106 | JOB FAILED: [#] LCCMInstallAgentJob(LCCMAgent(cluster-bound, ID:))
    com.vmware.eam.EamRemoteSystemException: Failed to get placement recommendation virtual machine in the cluster

Environment

VCF 9.0
VMware vSphere Kuberentes Service (VKS)

Cause

This failure is caused by a compounding sequence of placement constraints:

  1. Storage Policy Misconfiguration: A local VMFS datastore was incorrectly tagged or matched the rules for the Storage Policy assigned to the Supervisor deployment.

  2. EAM to DRS Conflict: EAM attempts to use the next available datastore matching the policy (the local VMFS datastore). DRS immediately rejects placing a cluster-level infrastructure VM on an unshared local datastore, resulting in a vim.fault.GenericDrsFault.

  3. If using VCF 9.0.0: You had ruled out the issue under: https://knowledge.broadcom.com/external/article/409163/supervisor-enablement-stuck-on-foundatio.html where it needs to be fixed as well.

Alternatively, the issue can occurdue to DRS PlaceVm API Limitation: The DRS placement engine algorithm has a hard limitation where it ignores datastores larger than 2 Petabytes (PB) (See KB 312187). The Trading_Dev_vSAN datastore exceeds 2 PB (approximately 2.8 PB as seen in capcaityBytes=2832551399915520). DRS drops this valid shared datastore from the placement candidate list.

Ultimately, the placement failure leaves an orphaned FLB deployment task (com.vmware.esx.settings.clusters.vms.solutions) locked in the vCenter Database, blocking the overall Supervisor enablement workflow.

Resolution

Correct the Storage Policy Tags for VMFS

  1. Navigate to Policies and Profiles > Storage Policies.

  2. Edit the Storage Policy associated with the Tanzu deployment.

  3. Review the Storage Compatibility tab.

  4. Ensure that no local VMFS datastores are listed as compatible. If they are, remove the assigned tags from those local datastores so they are not aevaluated by EAM.

  5. Restart the vCenter services or ideally reboot the vCenter.

If you are using datastores bigger than 2PB: You must bypass the size check for the target datastore.

  1. Identify the Managed Object ID (MoRef) of the vSAN datastore from vpxd.log (e.g., datastore-####).

  2. Open the vSphere Client and select the vCenter Server.

  3. Navigate to Configure > Advanced Settings and click Edit Settings.

  4. Add the MoRef ID to the advanced setting VmPlacementDatastoreOverride. (Note: If there are multiple datastores, separate them with commas).

  5. Click Add and save the configuration.

Additional Information

KB 312187: The DRS PlaceVm API does not work with datastores larger than 2PB

KB 409163: Supervisor enablement stuck on Foundation Load Balancer (FLB) deployment

 

Powered by Wolken Software