vCenter Server services fail to start due to high /storage/seat full

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server services cannot start as the /storage/seat partition is full due to large event tables.

 

Cause

Third-party software or monitoring tools may generate a large volume of data in the event tables.

Resolution

Please ensure there is a valid vCenter snapshot (Offline Snapshots in case of Linked Mode vCenters) before proceeding further.

1) Stop vpxd service.
    service-control --stop vmware-vpxd
 
2) Upload the attached clean_specific_events.sql file to vCenter into /tmp path, using WinSCP.
 
3) Modify the event types if any other event needs to be purged.

   lv_event_type1 = 'vim.event.UserLoginSessionEvent';
   lv_event_type2 = 'vim.event.UserLogoutSessionEvent';

   To list all events in VCDB, run: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"
 
4) Run the script. This will then create a new function called "remove_some_events" in VCDB.
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -f /tmp/clean_specific_events.sql
 
5) Validate if the newly created function exists in VCDB
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "\df remove_some_events"
 
6) Run the function to clear the events
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"
 
7) Reindex the database:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
   reindex database "VCDB";
 
8) Run a full vacuum in the database:
   vacuum full analyze verbose;
   vacuum full analyze;       -------- if the above command fails.
 
9) Exit the database:
   \q
 
10) Start the vpxd services:
    service-control --start vmware-vpxd



Note:

To periodically clear the event tables, you can call it using a script.

vi /storage/core/run_cleanup.sh

#!/bin/sh
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"

Change permissions
chmod 777 /storage/core/run_cleanup.sh

Schedule it (Example: To run at 2 am everyday) 
crontab -e
0  2 * * * /storage/core/run_cleanup.sh

 

Additional Information

1. While it's possible to clear all the event tables. However, this KB explores steps to clear specific events only.
2. Attached script is designed to specifically delete the below login and logout events.