Error: "The operation failed because no suitable resource was found." in VCD tenant UI every 2 hours for a replication

Products

• VMware Cloud Director

Issue/Introduction

• In VMware Cloud Director (VCD) you see errors every two hours saying:
  
  The operation failed because no suitable resource was found. Out of 1 candidate hubs: 1 hubs eliminated because: No valid storage container found for VIRTUAL_MACHINE_DISK "########-####-####-####-############" VmDiskRef [diskObjectId=########-####-####-####-############]. Out of # available storage containers only # were valid, and no solution could be computed across these remaining storage containers. Rejected hubs: resgroup-###### Storage containers were rejected for the following reasons: - PlacementException NO_FEASIBLE_PLACEMENT_SOLUTION"
  
  
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log in the VCD cells you can find errors similar to:
  
  | DEBUG    | task-service-activity-pool-1141 | ContainerState                 | Item 'VmDiskRef://EXISTING_DISK/########-####-####-####-############' requires 100,000 MB on container '
moref://########-####-####-####-############/Datastore#datastore-#####' | requestId=########-####-####-####-############,request=POST 
  ...
  | DEBUG    | task-service-activity-pool-1141 | ContainerState                 | Attempting to place item 'Item [itemId=VmDiskRef://EXISTING_DISK/########-####-####-####-############, itemSizeMb=10000, isThinProvisioned=false, itemIopsRequested=0]' on container 'Container [containerId=moref://########-####-####-####-############/Datastore#datastore-#####, usedCapacityMb=100000, uncommittedCapacityMb=1000, totalCapacityMb=200000, iopsCapacity=0, availableIops=0, isRemoteVsan=false]' | 
  
  
• The storage policy used for the replication in the destination site does not have thin provisioning enabled.

Environment

VMware Cloud Director 10.6

VMware Cloud Director Availability 4.7

Cause

Having thick provisioning enabled on the Storage policy makes VCD check for uncommitted space instead of free space when trying to allocate the new instance.

When VCDA creates a replication, in the destination Datastore, the VM disk VMDK is created as thick and the RTVM Disk is also created as thick, this makes the free and uncommitted spaces equal in the DS momentarily but then on first resize (after initial sync) it makes the RTVM disk thin which increases the free space on the disk, the size we see on the datastore as available, but the committed storage remains the same because the storage policy doesn't allow thin placement so when VCD tries to place a new instance there was not enough space uncommitted left in the DS to complete the action.

In the example visible in the LOG snippet in the Issue introduction we can see that there is a datastore with 200000MB in size which has occupied 100000MB leaving 100000MB of free storage, however the uncommitted space is only 1000MB which is not enough to accommodate the 10000MB requested.

Resolution

To resolve this issue enable thin provisioning on the storage policy, this way VCD when trying to place a new instance in the Datastore will check the free space instead of the uncommitted.