Error: "503 Service Unavailable" during long-running vCenter API jobs

Products

• VMware vCenter Server

Issue/Introduction

An automation tool or PowerShell script intermittently fails during execution with errors such as 503 Service Unavailable on the application side. The tool performs its operations using vCenter SOAP APIs.

Environment

vCenter 8.x

Cause

• The failures are caused by exhaustion of available SOAP/VAPI sessions on the vCenter Server during prolonged API activity.
• When the maximum number of concurrent sessions is reached, vCenter is unable to service new or existing API requests, resulting in intermittent SOAP API session drops.

Note:
The default maximum number of in-flight sessions is 3000, which is now shared between SOAP and vAPI sessions, as both use the same underlying session mechanism.

Resolution

Note: Take a snapshot of the vCenter Server before making any changes. If vCenter is configured in Linked Mode, take offline snapshots of all linked vCenter Servers simultaneously.

• Increase the maximum allowed session count on the vCenter Server by following the below steps,

1. 1. Log in to the vSphere Client as the Administrator
   2. Click on the vCenter Instance → Navigate to Configure → Advanced Settings.
   3. Click on Edit settings
   4. Add or modify the following advanced setting:
      • Name: config.vmacore.soap.maxSessionCount
      • Value: 6000 (or higher, depending on workload requirements)
   5. Save the changes.
   6. A restart of vCenter services is not required for these changes to take effect

Note: If the maxSessionCount is increased, then the APIs performance and scaling is no longer guaranteed - it may degrade. Also, the memory that the vAPI Endpoint and vxpd service will need increases, and thus it is recommended to also increase its memory, following the KB: Manually increasing the heap memory on vCenter Server Components