vCenter Server services fail to start due to full /storage/seat partition

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server services cannot start because the /storage/seat partition is full caused by large event tables in the VCDB.

• The size of the partition can be validated by executing the df -h command in the vCenter Server Appliance Command Line Interface.

• To list all events in VCDB, the following command is executed:

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

Third-party software or monitoring tools may generate a large volume of data in the event tables, filling the /storage/seat partition and causing vCenter Server services to fail.

Resolution

Important: Ensure there is a valid vCenter server snapshot (Offline Snapshots in case of Linked Mode vCenters) before proceeding further as per Snapshot Best practices for vCenter Server Virtual Machines.

The following steps need to be performed to purge specific high-volume events from the VCDB using a SQL script (clean_specific_events.sql) provided below in the attachments section.

1. SSH into the vCenter server as the root user to list all events in VCDB:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"

2. Modify the script event types depending on which event needs to be purged, attached clean_specific_events.sql script is specifically designed to delete login and logout session events in vim.event.UserLoginSessionEvent and vim.event.UserLogoutSessionEvent.

   lv_event_type1 = 'vim.event.UserLoginSessionEvent';  -------- Line 27 in the script.
lv_event_type2 = 'vim.event.UserLogoutSessionEvent'; -------- Line 28 in the script.

3. Stop vpxd service.
   service-control --stop vmware-vpxd


4. Upload the attached clean_specific_events.sql file into vCenter server /tmp directory using WinSCP software.
   

5. Run the script. This will then create a new function called "remove_some_events" in VCDB.
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -f /tmp/clean_specific_events.sql


6. Validate if the newly created function exists in VCDB.
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "\df remove_some_events"


7. Run the function to clear the events
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"


8. Reindex the database:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
reindex database "VCDB";


9. Run a full vacuum in the database:
   vacuum full analyze verbose;
vacuum full analyze;       -------- if the above command fails.
    
10. Exit the database:
    \q


11. Start the vpxd services:
    service-control --start vmware-vpxd


Note:

To periodically clear the event tables and prevent recurrence, the cleanup function can be called using an automated script.

1. Create a script file:
   vi /storage/core/run_cleanup.sh


2. Add the following content to the script:
   #!/bin/sh /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from remove_some_events ();"
   
   
3. Apply the execution permissions to the script:
   chmod 774 /storage/core/run_cleanup.sh


4. Schedule a cron job (Example: Daily execution at 2:00 AM):
   crontab -e
0  2 * * * /storage/core/run_cleanup.sh

Additional Information

While complete clearing of all event tables is possible, this document exclusively outlines the procedure for purging specific (vim.event.UserLoginSessionEvent and vim.event.UserLogoutSessionEvent) high-volume session events to maintain audit integrity where possible.