vCenter experiences high memory usage due to excessive ContainerView objects not being properly destroyed

Products

• VMware vCenter Server

Issue/Introduction

The vCenter Server (vpxd) service may experience:

• Heartbeat (HB) issues with ESXi hosts
• Increased memory consumption
• Performance degradation or slow response
• Intermittent UI/API latency
• vpxd service instability or unresponsiveness
• vpxd crash or restart

Such behaviors can occur when a large number of ContainerView objects are created via the vSphere API but are not properly destroyed.

Environment

• VMware vCenter Server (all versions)
• Environments using:
  • PowerCLI / pyvmomi / SDK-based automation
  • Third-party tools (backup, monitoring, orchestration)
  • Custom scripts or integrations

 

Cause

Each createContainerView API call creates a server-side view object within vpxd.

• These objects consume memory and session resources
• They must be explicitly destroyed

If a client repeatedly creates ContainerView objects without properly destroying them:

ContainerView objects accumulate in memory → leading to memory exhaustion in vpxd

Resolution

To identify potential ContainerView leaks, vpxd logs can be analyzed to compare:

• Number of vim.view.ViewManager.createContainerView operations
• Number of view.destroy operations

A mismatch indicates potential leakage.

The username and client IP address information are recorded in the vpxd-profile logs, which can be correlated with vpxd.log entries for detailed analysis.

A custom script ViewLeakTracer.sh can be used to aggregate ContainerView usage per username and client IP address, helping identify the source of the leak.

To run the script,

1. Upload the script ViewLeakTracer.sh to the vCenter Server using SFTP.
   For example, place it under /tmp or /root.
2. SSH to the vCenter
3. Grant execute permission to the script,
   chmod +x <path>/ViewLeakTracer.sh
4. Navigate to the vpxd log directory,
   cd /var/log/vmware/vpxd
5. Note: It is not required to extract vpxd*.log.gz or vpxd-profiler*.log.gz.
6. Execute the script and wait for 1 or two minutes,
   /<path>/ViewLeakTracer.sh

Sample Output

vCenter ContainerView Leak Report (Sorted by Net_Leak)

Username                                                Client_IP        Total_Create    Total_Destroy   Net_Leak(Diff)
###\###                                                 ###.###.###.###  XXX             YYY             (XXX-YYY)
------------------------------------------------------------------------------------------------------------------------
###\###                                                 ###.###.###.###  ZZZ             ZZZ             0

• Total_Create → Number of ContainerView objects created
• Total_Destroy → Number of ContainerView objects destroyed
• Net_Leak (Diff) → Difference between create and destroy

A positive Net_Leak indicates potential unreleased ContainerView object.

Additional Information

• Entries with higher Net_Leak values are primary suspects
• Internal services (e.g., vpxd-extension, vsan.health) usually have minimal differences
• Larger gaps often indicate:
  • Misbehaving scripts
  • Third-party tools
  • Improper API usage

• Entries showing Unknown_User / Unknown_IP in the script output indicate that the ContainerView operation could not be correlated between the vpxd.log and vpxd-profiler.log.

  This typically occurs when the log time ranges do not overlap. If a ContainerView operation is present in the vpxd.log but not found in the vpxd-profiler.log, it is reported as Unknown_User / Unknown_IP.