CPU utilization of the vpxd.exe process on vCenter server is constantly 100% for large environments

Products

• VMware vCenter Server

Issue/Introduction

This article provides information for:

• Customers with 200+ hosts and 2000+ virtual machines.
• Customers that have upgraded from vCenter Server 2.5 or vCenter Server 4.0 and are now on vCenter Server 4.1 or above.

Symptoms:

• Slow performance in the vSphere Client.
• vpxd.exe consistently running at or near 100% CPU utilization.
• You are upgrading your vCenter Server.
• When performing normal operations such as editing a virtual machine or performing a vMotion, you see an error similar to:
  
  The request failed because the remote server 'vcenter_server' took too long to respond. (The command has timed out as the remote server is taking too long to respond.)
  
  
  

Environment

VMware vCenter Server 4.1.x
VMware vCenter Server 5.0.x

Cause

This issue occurs because during the upgrade, the vpxd.CacheSize parameter was not appropriately increased.

Resolution

Confirm that you are experiencing this issue before attempting to complete the steps outlined in the resolution below.

To confirm that you are experiencing this issue:

1. Open a browser window and go to https://vCenter_Server/VOD/index.html.
2. In Detail Pages, click Pull Counters.
3. Look for the string Evictions and check to see if any counters are non-zero.
   
   Note: There are approximately 14 categories of Evictions. Look for all occurrences of Evictions and ensure that all instances of .../Evictions/Count/total are at zero. These lines are spread throughout the pull counters page, rather than appearing in one big group as shown here.
   
   If you are hitting this issue, the counters are non-zero:
   
   /InventoryStats/VpxdCache/Nmspc='resConfig'/Evictions/Count/total 19568
   /InventoryStats/VpxdCache/Nmspc='vmconfig'/Evictions/Count/total 3569418
   
   If you are not hitting this issue, the counters are zero:
   
   /InventoryStats/VpxdCache/Nmspc='resConfig'/Evictions/Count/total 0
   /InventoryStats/VpxdCache/Nmspc='vmconfig'/Evictions/Count/total 0

To resolve this issue:

1. Log into the vSphere Client.
2. Go to Administration > vCenter Server Settings > Advanced Settings > VirtualCenter.CacheSize.
3. Change the value to 9223372036854775807 (unlimited).
4. You can also verify that this value is set by looking for this entry in the vpxd-profiler-#.log. However, instead of showing the above value for unlimited, the profiler logs will show a value of -1.
   
   /InventoryStats/VpxdCache/Capacity/total -1
   
   
5. In the task manager, the vpxd.exe process shows a drop in CPU utilization almost immediately.

Note:

• This value can be changed without a VirtualCenter restart.
• When you change this parameter, the vpxd memory utilization may increase.
• Even though the CPU utilization shows a drop almost immediately, these counters do not automatically go to zero after you apply the changed value. Instead, they stop increasing. If you restart the vCenter Server service, they drop down to zero.

Additional Information

How to stop, start, or restart vCenter Server services
大規模環境で vCenter Server の vpxd.exe プロセスの CPU 使用率が常に 100% になる
在大型环境中，vCenter Server 上的 vpxd.exe 进程的 CPU 利用率持续为 100%

Impact/Risks: