vCenter Server services fail to start due to full /storage/seat partition

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server services cannot start because the /storage/seat partition is full caused by large event tables in the VCDB.
  
  
• The size of the partition can be validated by executing the df -h command in the vCenter Server Appliance Command Line Interface.
  
  
• To list all events in VCDB, the following command is executed:

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"

Environment

VMware vCenter Server

Cause

Third-party software or monitoring tools may generate a large volume of data in the event tables, filling the /storage/seat partition and causing vCenter Server services to fail.

Resolution

Please ensure there is a valid vCenter snapshot (Offline Snapshots in case of Linked Mode vCenters) before proceeding further as per Snapshot Best practices for vCenter Server Virtual Machines.

The following steps are performed to purge specific high-volume events from the VCDB using a provided SQL script.

1. Stop vpxd service.
   service-control --stop vmware-vpxd


2. Upload the attached clean_specific_events.sql file to vCenter into /tmp path, using WinSCP.
   
   
3. Modify the event types if any other event needs to be purged.
   
   lv_event_type1 = 'vim.event.UserLoginSessionEvent';
lv_event_type2 = 'vim.event.UserLogoutSessionEvent';
   
   To list all events in VCDB, run: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"


4. Run the script. This will then create a new function called "remove_some_events" in VCDB.
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -f /tmp/clean_specific_events.sql


5. Validate if the newly created function exists in VCDB
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "\df remove_some_events"


6. Run the function to clear the events
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"


7. Reindex the database:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
reindex database "VCDB";


8. Run a full vacuum in the database:
   vacuum full analyze verbose;
vacuum full analyze;       -------- if the above command fails.
    
9. Exit the database:
   \q


10. Start the vpxd services:
    service-control --start vmware-vpxd


Note:

To periodically clear the event tables and prevent recurrence, the cleanup function can be called using an automated script.

1. A script file is created:
   vi /storage/core/run_cleanup.sh


2. The following content is added to the script:
   #!/bin/sh /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"
   
   
3. Execution permissions are applied to the script:
   chmod 777 /storage/core/run_cleanup.sh


4. A cron job is scheduled (Example: Daily execution at 2:00 AM):
   crontab -e
0  2 * * * /storage/core/run_cleanup.sh

Additional Information

• While complete clearing of all event tables is possible, this document exclusively outlines the procedure for purging specific, high-volume events to maintain audit integrity where possible.
  
  
• The attached clean_specific_events.sql script is specifically designed to delete login and logout session events (vim.event.UserLoginSessionEvent and vim.event.UserLogoutSessionEvent).