"A specified parameter was not correct: key" error when resizing a single disk on a virtual machine in VMware Cloud Director versions 10.4 and 10.4.1.

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• "A specified parameter was not correct: key" error when resizing a single disk on a virtual machine in VMware Cloud Director versions 10.4 and 10.4.1.
• The storage policy is set to unlimited, so no quotas are applied.
• The backend datastore provides enough space for the resizing operation.
• The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log shows errors of the form:

| ERROR    | task-service-activity-pool-3855 | SdrsApiInvocationManagerImpl   | SDRS cancelRecommendation error: A specified parameter was not correct: key | requestId=<REQUEST_UUID>,request=POST https://vcloud.example.com/api/vApp/vm-<VM_UUID>/action/reconf...,requestTime=1668518510774,remoteAddress=<IP>:56789,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 38.0.0-alpha vcd=<VDC_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>)
(vmodl.fault.InvalidArgument) {
   faultCause = null,
   faultMessage = null,
   invalidProperty = key
}

Environment

VMware Cloud Director 10.4

VMware Cloud Director 10.4.1

Cause

This issue can occur if Cloud Director requests that a recommendationKey key is removed in vCenter but this recommendationKey key no longer exists in vCenter.

Such recommendationKey keys are part of SDRS reservations which are either timed out automatically by vCenter or cleared on request.
If Cloud Director believes that a key is still present but vCenter has already removed it then this error can occur.

Resolution

This is a Known Issue in VMware Cloud Director 10.4 and 10.4.1.
The issue resolved in 10.4.2-GA

 If you are unable to upgrade to 10.4.2, you can leverage the workaround below:

Workaround:

To workaround this issue the stale reservations referencing the recommendationKey key must be removed from the Cloud Director Database.

1. SSH to the primary cell
2. Run a database backup: /opt/vmware/appliance/bin/create-db-backup.sh
3. Log into the DB: sudo -u postgres psql vcloud
4. Run the following:

select * from sdrs_reservation;
select * from reservation;

5. Delete any stale entries that show. Example:

Delete from sdrs_reservation;
Delete from reservation;

This is not a permanent workaround, and should the issue rise again, the same steps must be followed.