VMware vCenter Server resource usage is 100% and becomes unresponsive

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

The vpxd.exe process consumes 100% of the CPU resources.
The vpxd.exe process consumes large amount of the physical memory.
You cannot log into VMware vCenter Server.
In the performance chart for the cluster view, you see the message:

The data source timed out
You see the error:

The request failed because the remote server took too long to respond
In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log file, you see calls to the vCenter Server PerformanceManager API multiple times per second that appear similar to:

[2012-07-17 09:47:24.861 13880 info 'App' opID=51d096b0] [VpxLRO] -- BEGIN task-internal-19566612 --
-- vim.PerformanceManager.queryStats -- A5113EB8-####-####-####-########D19(374B3D36-####-####-####-########DDD)

Environment

VMware vCenter Server 4.1.x
VMware vSphere ESXi 5.5
VMware vCenter Server 5.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server 4.0.x
VMware vCenter Server 5.1.x

Cause

Excessive calls to the PerformanceManager API cause vpxd.exe to consume a large number of resources. These calls are extremely resource intensive as they generate statistics on the fly and are highly computational.

Some third-party monitoring products generate excessive calls to this API and drive up the resource load to unsustainable levels.

Resolution

The session ID observed in the vpxd.log file (in these examples, A5113EB8-C947-4D5D-A477-B9CFB084AD19) can be used to search the vpxd-profiler.log file to determine the offending user or service account name, as well as the originating IP address.

To resolve this issue:

Search for the session ID in the vpxd-profiler.log file. This reveals an additional information about the origin of the session causing the resource load on the PerformanceManager interface.

For example:

B9CFB084AD19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/NullCollectorCount/total 0
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/QueuedOpsCount/total 0
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/ReadLocked/total 0
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/TotalObjectCount/total 3239
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/TriggeredFiltersCount/total 1
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/PropertyCollector/TriggeredProcessGUReqs/total 0
/SessionStats/SessionPool/Session/Id='A5113EB8-####-####-####-########D19'/Username='DOMAIN\UserName'/ClientIP='10.0.0.1'/SoapSession/Id='11808511'/Count/total 1

In this example, the account used to call the PerformanceManager API was DOMAIN\UserName, and it originated from a workstation with an IP of 10.0.0.1.
In the vpx-profiler.log file, ensure that the CPU usage tracking counter is running at 100%. Locate lines which repeat approximately every 5 minutes which are similar to:

/ProcessStats/ProcessorCpuUsage/mean 100
- In vCenter 5.x, this value is expressed as a percentage multiplied by 100. Therefore 100% is represented by a value of 10000.
- In vCenter 4.1 and earlier, this value is expressed as a percentage only, therefore 100% is represented by a value of 100.
After you determine the user or service account and the origin of the connection, disable the account or service to prevent it from connecting to vCenter Server.
After disabling the account or service, verify that resource usage returns to acceptable levels.

Note: If the issue is found to be with a third-party product, contact the vendor for additional support.