Unable to deploy/modify VMs after upgrading to Cloud Director version 10.4.2 or later

Products

• VMware Cloud Director

Issue/Introduction

This document is designed to restore functionality to a VMware Cloud Director environment after upgrading to version 10.4.2 or later and encountering errors with VM placement

• After upgrading VMware Cloud Director to version 10.4.2, users are no longer able to deploy/modify VMs in the environment
• The error indicates there is no feasible placement solution because all storage containers have been rejected for placement
• Receiving errors similar to the following when trying to modify/deploy VMs:

The operation failed because no suitable resource was found. Out of 1 candidate hubs: 1 hubs eliminated because: No valid storage containers found for VirtualMachine "<VM_UUID>". All 3 available storage containers were filtered out as being invalid. Rejected hubs: resgroup-123 Storage containers were rejected for the following reasons: 1 storage containers rejected because: Threshold would be violated. Rejected storage containers: DatastoreCluster - PlacementException NO_FEASIBLE_PLACEMENT_SOLUTION

Environment

VMware Cloud Director 10.4.2

VMware Cloud Director 

Cause

• This issue is the result of a code fix implemented in version 10.4.2 that resolves a bug in previous versions where red and yellow thresholds were not applied on individual datastores in a datastore cluster
• In version 10.4.2, red and yellow thresholds are applied to individual datastores in a datastore cluster
• Previously, if you set thresholds on a datastore cluster, VMware Cloud Director did not apply the thresholds on each individual datastore in the cluster. As a result, the placement engine could place virtual machines on the datastores even after the threshold is breached
• If you configure the thresholds for new datastore clusters, VMware Cloud Director sets the threshold value on each individual datastore and not as an aggregation on the datastore cluster level

Resolution

After upgrading to VMware Cloud Director 10.4.2, for every pre-existing datastore cluster, you must divide the existing threshold to the number of datastores in that cluster

Workaround:

• To identify this issue during troubleshooting, reduce the red threshold to a value that is readily available on each datastore in the datastore cluster then attempt to deploy a VM. If the placement engine does not generate any errors, you'll know that this was indeed causing the issue. After identifying the root cause, you can now work to determine an adequate red threshold value specific to the environment

Additional Information

• VMware Cloud Director 10.4.2 Release Notes:
   

• About the Workload Placement Engine:

https://techdocs.broadcom.com/us/en/vmware-cis/cloud-director/vmware-cloud-director/10-5/vmware-cloud-director-service-provider-portal-guide-10-5/managing-provider-vdcs-admin/managing-the-resource-pools-on-a-pvdc-admin/about-the-workload-placement-engine-admin.html

• Configure Low Disk Space Thresholds for a Provider Virtual Data Center Storage Container:

https://techdocs.broadcom.com/us/en/vmware-cis/cloud-director/vmware-cloud-director/10-5/configure-low-disk-space-thresholds-for-a-provider-vdc-datastore.html

• View the Datastores and Datastore Clusters on a Provider Virtual Data Center:

https://techdocs.broadcom.com/us/en/vmware-cis/cloud-director/vmware-cloud-director/10-5/view-the-datastores-and-datastore-clusters-in-pvdc.html

Impact/Risks:

• This can prevent deployment of new VMs as well as the modification of existing VMs