Error: "Total available resources could not be determined for reservation" when changing Storage Policies in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When relocating a VM from one storage policy to another you see the error
   
  Total available resources could not be determined for reservation
• in /opt/vmware/vcloud-director/logs/vcloud-container-debug.log in the VCD cell you see entries similar to:
  

  2019-01-01 05:30:14,828 | WARN     | pool-jetty-8012636        | BatchTask                      | Error executing backend call. | requestId=<REQUEST_UUID>,request=POST https://vcdhostname/cloud/amfsecure,requestTime=1554370204726,remoteAddress=<IP>:52041,userAgent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (...,accept=*/* method=vAppService.updateVAppVmWithStorageClasscom.vmware.vcloud.common.reservation.ReservationSystemException: Total available resources could not be determined for reservation

          at com.vmware.vcloud.common.reservation.dao.impl.ReservationDaoImpl.getTotalAvailable(ReservationDaoImpl.java:389)
        at com.vmware.vcloud.common.reservation.dao.impl.ReservationDaoImpl.getTotalAvailable(ReservationDaoImpl.java:401)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:205)

 

Environment

VMware Cloud Director 10.x

Cause

This issue occurs when Cloud Director's reservation system encounters a timeout.
By default the timeout is set to 10 seconds.

Resolution

As a first step try a reconnect of the vCenter in Cloud Director to see if it resolves the issue. 

If reconnecting the vCenter doesn't help follow this steps:

1. SSH to the primary VCD cell and login with the root user
2. increase the vcloud.reservation.callback.timeoutInSeconds property
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.reservation.callback.timeoutInSeconds -v 50
3. Restart the vmware-vcd service for the changes to take effect
   systemctl restart vmware-vcd

For more information on using this subcommand, see Updating Application Configuration Settings.

Workaround:

To workaround this issue without changing the timeout, set the storage limit to Unlimited for the VDC storage policy.

1. Navigate to the Organization
2. Select the affected Org VDC
3. Navigate to Storage Policies
4. Right-click the affected Storage Policy and select Properties
5. Select Unlimited

Additional Information

In Cloud Director 9.7 a new configuration parameter called "vcloud.reservation.callback.timeoutInSeconds" has been added which allows this timeout to be increased.

In previous versions of Cloud Director it is not possible to change this timeout.

Configuration Parameters such as the above can be added to Cloud Director via the Cell-Management-Tool command on the Cells and using the manage-config subcommand to update the application configuration settings.