Scheduled vCenter Server backup fails due to memory exhaustion by vc-ws1a-broker

Products

• VMware vCenter Server

Issue/Introduction

• Scheduled file-based backup jobs fail on the vCenter Server.

• A backup job failure alarm is triggered in the vSphere Client.

• Following message can be encountered in /var/log/vmware/applmgmt/backup.log:
  

  YYYY-MM-DDThh:mm:ss.xxx [########-######-########] [MainProcess:PID-######] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Hit exception inside process StellarDbBackup: [Errno 12] Cannot allocate memory: '/usr/bin/python

Environment

VMware vCenter Server

Cause

The backup process fails during the initiation phase (fork process) because it cannot allocate sufficient OS virtual memory.

This virtual memory exhaustion is primarily caused by the vc-ws1a-broker service consuming an excessive amount of memory, pushing the overall OS memory request close to 100% capacity.

When the backup process attempts to run, the system is unable to provide the necessary memory, resulting in the failure.

Resolution

To prevent the vc-ws1a-broker service from exhausting memory and causing future backup failures, apply the resolution steps outlined in the following Knowledge Base article:

• Login to vCenter Server with External Identity Providers for SSO account stops working

Note: Although KB 378474 describes a different symptom (SSO login failure), the underlying cause (vc-ws1a-broker memory consumption) and the required resolution steps are identical. Follow the instructions in the "Resolution" section of that article.