vpxd service on vCenter Server crashes with error message "Current value exceeds hard limit. Shutting down process" in vpxd.log file

Products

VMware vCenter Server

Issue/Introduction

vpxd service keeps on crashing on vCenter Server frequently with the following error message within the vpxd logs -

Log Location - /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS error vpxd[634618] [Originator@6876 sub=Memory checker] Current value <Current_Used_Memory> exceeds hard limit <Assigned_Memory_to_vCenter>. Shutting down process.
YYYY-MM-DDTHH:MM:SS panic vpxd[634618] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: <vCenter_Server_version>, build: build-<build_number_of_vcenter>, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00531DD5]
--> backtrace[01] libvmacore.so[0x00421834]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00434013]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x0050A993]
--> backtrace[04] libvmacore.so[0x0050AAAB]: Vmacore::PanicExit(char const*)
--> backtrace[05] libvmacore.so[0x00421556]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[06] libvmacore.so[0x00385107]
--> backtrace[07] libvmacore.so[0x0037EC04]
--> backtrace[08] libvmacore.so[0x00384517]
--> backtrace[09] libvmacore.so[0x00510FC5]
--> backtrace[10] libpthread.so.0[0x00008EB0]
--> backtrace[11] libc.so.6[0x000FFADF]
--> backtrace[12] (no module)
--> [backtrace end]

A vpxd core dump file is dumped under /var/core directory with the following name - core.vpxd-worker.#####
Statistics settings of vCenter Server is set to Level 2 or higher on vCenter Server when checked under vCenter Server instance > Configure > Settings > General > Statistics :
Running "vimtop" command on the vCenter Server's SSH shows high vpxd memory utilization.

Database size of the vCenter Server is reported as high with the following tables occupying the highest space - "vc.vpx_device" and "vc.vpx_stat_counter". The same can be confirmed by logging in to the vCenter Server database and checking the table size information.
Open an SSH session to the vCenter Server and run the following command to interact with the vCenter Server database and check the top 10 table by size:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 10;

Confirm if top two largest table by size are vc.vpx_device" and "vc.vpx_stat_counter" :

             relation              | total_size 
-----------------------------------+------------
 vc.vpx_device                     | 27 GB
 vc.vpx_stat_counter               | 9555 MB
 vc.vpx_usage_stat                 | 1912 MB
 vc.vpx_topn_past_day              | 644 MB
 vc.vpx_journal_entry              | 625 MB
 vc.vpx_vm                         | 513 MB
 vc.vpx_host                       | 458 MB
 vc.vpx_hist_stat4_41              | 430 MB
 vc.vpx_hist_stat4_40              | 428 MB
 vc.vpx_hist_stat4_39              | 426 MB

Environment

VMware vCenter Server

Cause

Issue happens as the tables within the database are storing the login events from third party applications like backup tool, etc and not getting cleared making vpxd to consume more memory and ultimately crashing.

Resolution

Ensure to set the Statistics Level on vCenter Server to Level 1 for long term. Document for reference - Configuring Statistics Settings

To clear the events from the above mentioned tables within vCenter Server's database, follow the below steps :

Take a Snapshot of vCenter Server VM.
Open an SSH session and stop the vpxd service -
```
service-control --stop vpxd
```
Login to the vCenter Server Database with the following command -
```
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
```

Run the following commands to clear the entries from the top two consuming tables

1. delete from vpx_stat_counter where device_id in(select device_id from vpx_device where device_name like '%/SessionStats/SessionPool/Session%'); 

2. delete from vpx_device where device_name like '%/SessionStats/SessionPool/Session%'; 

3. vacuum  (full, ANALYZE , verbose)  vpx_device; 

4. vacuum  (full, ANALYZE , verbose) vpx_stat_counter;

Restart the vpxd service -
```
service-control --start vpxd
```
Check the memory utilization of the vpxd service again with "vimtop" command and ensure that the memory utilization is not increasing again.