vCenter Server unresponsive due to core dumps consuming all available disk space

Products

• VMware vCenter Server

Issue/Introduction

The vCenter Server Appliance (VCSA) becomes unresponsive or inaccessible. Management services fail to start, and the /storage/core or /storage/log partitions are at 100% capacity.

Symptoms:

• vCenter Server UI and API are unresponsive.
• Large core files (e.g., core.updatemgr-worker.####) fill the disk space.
• The vmware-vum-server.log shows thousands of entries per hour with the following error: com.vmware.vcIntegrity.lifecycle.EsxImage.AddonBaseImageMismatchError
• Log messages indicate an incompatible Add-on: Addon [Addon-Name] is incompatible with base image version [ESXi-Version].

Environment

• VMware vCenter Server 7.0.x / 8.0.x / 9.x
• vSphere Lifecycle Manager (vLCM)

 

Cause

This issue occurs when an incompatible vendor add-on is present in the vSphere Lifecycle Manager depot. The vLCM service continuously attempts to validate the image, failing repeatedly and generating excessive logs and core dumps that eventually exhaust the appliance's disk space.

Resolution

To resolve this issue, the Lifecycle Manager database must be reset to clear the invalid metadata and stop the logging loop.

1. Identify and remove core dumps:
   • Connect to the VCSA via SSH.
   • Navigate to the core directory: cd /storage/core
   • Identify and remove large core files to restore temporary service access: rm core.updatemgr-worker.####
2. Snapshot vCenter:
   • Take an offline (powered-off) snapshot of the vCenter Server.
3. Reset the VMware Update Manager Database:
   • Follow the steps in KB 316581 to perform a full reset of the Update Manager Database.
   • Note: This is a destructive task. Custom baselines and manually imported patches must be reapplied.
4. Reconfigure Download Sources:
   • Re-add any custom URLs or Broadcom tokenized URLs.
   • Initiate a Full Sync in the Lifecycle Manager UI.
5. Validate Image:
   • Ensure the target host/cluster image no longer contains the incompatible add-on.