"Internal Server Error" when attempting to Migrate Tenant Storage in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Using the Migrate Tenant Storage option in the Cloud Director UI to migrate all the items stored on a datastore to other datastores in an SDRS Cluster.
• The migration attempt fails to migrate Virtual Machines with errors of the form:

Internal Server Error
Caused by: java.lang.RuntimeException: The operation failed because no suitable resource was found. Out of x candidate hubs:
x hubs eliminated because: No valid storage containers found for VirtualMachine "{vm-uuid}". All x available storage containers were filtered out as being invalid. Rejected hubs: resgroup-XX

Storage containers were rejected for the following reasons:
 x storage containers rejected because: They are disabled.  Rejected storage containers: {datastores}
 y storage containers rejected because: They do not have enough storage capacity.  Rejected storage containers: SDRS_cluster
 z storage containers rejected because: They are not in the list of required datastores [[moref://datastorepath]].  Rejected storage containers: {datastores}
 

Environment

VMware Cloud Director 10.x

Cause

This is a known issue in VMware Cloud Director 10.3.

Resolution

This issue is resolved in VMware Cloud Director 10.4 onwards available at VMware Downloads .

Workaround:

To workaround this issue the VMs can be migrated off the datastore to be decommissioned by changing their VM Storage Policy and not by utilising the Migrate Tenant Storage option.

Example steps would be as follows:

1. Identify the name of the datastore which is to be decommissioned, for example "DatastoreD".
2. Identify the name of the datastores which are to be retained and to which the VMs are to be migrated, for example "DatastoreA", "DatastoreB", "DatastoreC".
3. Create a new/temporary VM Storage Policy in vSphere which contains the datastores to be migrated to, for example "StoragePolicy1"
   Ensure that the Storage Compatibility tab of the new VM Storage Policy only shows the datastores to which we wish to migrate the VMs, for example "DatastoreA", "DatastoreB", "DatastoreC".
   For more information on creating Storage Policies see the vSphere documentation here, Create a VM Storage Policy for Tag-Based Placement .
   NOTE: datastores may be part of multiple Storage Policies and so do not need to be removed from the original Storage Policy.
4. In the Provider UI of Cloud Director Add this new Storage Policy to the Provider VDC backing the Organization VDC containing VMs to be migrated.
   This can be done under Cloud Resources > Provider VDCs > Provider VDC Name > Policies > Storage.
5. Next Add this Storage Policy to the Organization VDC where the VMs to be migrated are located.
   This can be done under Cloud Resources > Organization VDCs > Organization VDC Name > Policies > Storage.
6. In the Tenant UI of Cloud Director click on the name of a VM which failed to be migrated.
7. Click on the General > Edit option and change the Storage Policy of the VM from the current one to the new Storage Policy which contains the datastores to which we wish to move, for example "StoragePolicy1".
   Click Save and the VM will migrate to one of the available datastores.
8. After this migration has completed check the Hardware > Hard Disks section of the VM.
   For any Hard Disk which does NOT use the VM default policy we may need to also change its Policy setting to the new Storage Policy if it still remains on the datastore to be decommissioned.
9. Repeat the steps 6. to 8. for any VM located on the datastore to be decommissioned.
10. Once all VMs have been updated retry the Migrate Tenant Storage action from Cloud Director on the datastore to be decommissioned and confirm that this now succeeds.
    This will ensure that no further Cloud Director objects remain on that datastore and it can be removed from the old/original Storage Policy at the vSphere level so that Cloud Director no longer utilises it.
11. With the datastore decommissioned successfully we can set the VMs which were migrated to the new Storage Policy back to their original Storage Policy if desired.
    The new/temporary Storage Policy can subsequently be removed from the Organization VDC, Provider VDC, and then vSphere.