VCSA VMware vCenter Service (“vpxd”) randomly stopping | vpxd crashing due to 3rd party tool querying

Products

• VMware vCenter Server

Issue/Introduction

After manually starting the "vpxd" service it runs for a while and then crashes again.

Log file for the the vCenter Server service (var/log/vmware/vpxd/vpxd-*log) from the crashed session indicates the memory usage exceeded the hard limit.

error vpxd[nnnnnn] [Originator@6876 sub=Memory checker] Current value 22352640 exceeds hard limit 22156288. Shutting down process.
panic vpxd[nnnnnn] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic

Cause

Increase in transient memory usage caused by multiple/parallel API calls collecting performance statistics (vim.PerformanceManager.queryStats) 

• One or more client(s) running 1000s of parallel API calls to collect performance statistics for inventory objects.
  • The data required for these calls has be to fetched from database and retained in memory till transaction is complete.
• Several such requests are issued in parallel resulting in significant spike of vpxd memory consumption. 
  • When usage hits the high watermark (hard limit), memory checker stops the service.

Resolution

Suggestions:

1. Modify the clients to either reduce the request rate, or modify QuerySpec to return a smaller result set.
   1. Disable the clients till you update the program/script on those clients.
2. Alternatively,  you can increase the memory limit of vpxd to a higher value following steps below:
   cloudvm-ram-size -C <sizeInMB> vmware-vpxd  #(Increases the memory limit.)
   vmon-cli -r vpxd #(Restarts vpxd for the setting to take effect)
   1. Note that increasing the maximum limit depends on overall availability of memory resources on the appliance and may come at the expense of other services. 

Additional Information

• VMware vCenter API PerformanceBest Practices