Error "Too many outstanding operations" in VMware Cloud Director during deployment or provisioning

Products

• VMware Cloud Director

Issue/Introduction

VMware Cloud Director (VCD) operations, such as cell deployment or VM provisioning, fail with a specific exception.

Symptoms:

• VCD UI or logs report: com.vmware.vcloud.api.rest.exceptions.VCloudException: Too many outstanding operations.

• vCenter Server Task Console shows a high volume of pending or "In Progress" tasks.

• Tasks may be originated by a 3rd-party backup solution or management plugin and appear stuck for multiple days.

• New tasks initiated by VCD remain in a "Queued" state or fail immediately.

Environment

10.x

Cause

The vCenter Server has reached its maximum threshold for concurrent tasks. This is typically caused by orphaned or hung API sessions from 3rd-party integrations (e.g., backup software) that fail to close tasks. When the task limit is reached, the vCenter API rejects incoming requests from VCD, resulting in the "Too many outstanding operations" error.

Resolution

To resolve this issue, the vCenter task queue must be cleared.

1. Identify and Cancel Stuck Tasks:

   • Log in to the vSphere Client.

   • Go to Monitor > Tasks.

   • Filter for tasks that have been running for an abnormal duration (e.g., > 24 hours).

   • Right-click and select Cancel on these orphaned tasks.

2. Restart vCenter Services (If tasks cannot be cancelled):

   • If the tasks remain in the UI after cancellation, the vpxd service must be restarted to clear the in-memory task queue.

   • Connect to the vCenter Server Appliance via SSH.

   • Run the following command: service-control --restart vmware-vpxd

3. Optimize 3rd-Party Integration:

   • Review the backup solution's configuration to ensure it is not exceeding the recommended concurrent snapshot limit.

   • Verify that the backup service account has appropriate permissions and is not hitting session limits.

Additional Information

For more information on vCenter Server limits, refer to the VMware Configuration Maximums tool.