Commvault Backup jobs are failing when the backup tries to establish a HTTP service request to the vCenter Server

Products

VMware vCenter Server

Issue/Introduction

Backup jobs for external backup applications (e.g. Commvault backup) are failing while due to the backup application being unable to establish HTTP communication with the vCenter server

The logs of the backup proxy (e.g. vsbkp.log for Commvault) might contain messages similar to below:

<timestamps><thread> _Connect() - Exception - System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)at Vim25Api.VimService.RetrieveServiceContent(ManagedObjectReference _this)    at VISDKWrapper.SvcConnection._Connect(String url, String username, String password)
##### #### <timestamp><thread> 9 VSBkpCoordinator::OnMsgVMBackupGetNextVM
##### #### <timestamp><thread>  CVMWareInfo::Connect() - VISDKCppBridge::Connect failed [The request failed with HTTP status 503: Service Unavailable.

Upon reviewing the vCenter service logs (/var/log/vmware/vpxd/vpxd-<number>.log) in the vCenter Server Appliance, messages similar to the following are seen:

YYYY-MM-DDTHH:MM:SS error vpxd[7F99DECD9700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
YYYY-MM-DDTHH:MM:SS error vpxd[7F99D7468700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
YYYY-MM-DDTHH:MM:SS info vpxd[7F99DF2E5700] [Originator@6876 sub=vpxLro opID=########] [VpxLRO] -- BEGIN lro-28950290 -- session[<sessionID>]<sessionID> -- vim.HttpNfcLease.progress -- <vpxsessionID>(<ssosessionID>)
YYYY-MM-DDTHH:MM:SS info vpxd[7F99D5D3A700] [Originator@6876 sub=vpxLro opID=########] [VpxLRO] -- BEGIN lro-28950648 -- session[<sessionID>]<sessionID> -- vim.HttpNfcLease.progress -- <vpxsessionID>(<ssosessionID>)
YYYY-MM-DDTHH:MM:SS error vpxd[7F99E7FFF700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.x

Cause

This issue occurs when the currently active SOAP sessions exceed 2000, which is the limit for concurrently open HTTP sessions in vpxd. SOAP connections can be generated by VMware vSphere Client sessions, Third-party API connections, vCenter Server plugins.

Resolution

To work around vCenter backup or connection issues caused by session limitations by increasing the maximum allowed SOAP sessions in vpxd.cfg.

Pre-requisite: Backup Before Making Changes

Before modifying any configuration files, ensure you do the following:

Take a Snapshot of the vCenter Appliance
Back up the vpxd.cfg file
- Log in to the vCenter Appliance via SSH or console.
- Backup the current configuration:
  
  cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

Steps

Stop the vpxd Service

service-control --stop vpxd
Edit the vpxd.cfg File
- Open the file with a text editor (e.g., vi or nano):
  
  vi /etc/vmware-vpx/vpxd.cfg
- If the <soap> and If <maxSessionCount> elements are not already present, you need to add them manually under the <vmacore> block.
- Add the following entries:
```
<config>
  ...
  <vmacore>
    ...
    <soap>
      <maxSessionCount>6000</maxSessionCount>
    </soap>
    ...
  </vmacore>
  ...
</config>
```
NOTE : These entries will not be pre-populated; you need to insert them manually.
Save the File and Exit the Editor
- In vi, press Esc, type :wq!, and press Enter to save and exit.
Start the vpxd Service

service-control --start vpxd

Note: Please be aware that the number of concurrent HTTP sessions for vpxd is limited because a high count of open sessions can lead to high resource consumption for vpxd, thus potentially causing resource shortages.

Additional Information

Similar related information can be found in the articles mentioned below :