vCenter service fails to start due to /storage/seat/ partition is at 100% full on vCenter Server Appliance 8.U3e.

Products

• VMware vCenter Server 8.0

Issue/Introduction

• df -h displays the following  /storage/seat/ partition is at 100%
• vCenter services fails to start.
• Unable to run vacuumdb or truncate table : 
  • On running : /opt/vmware/vpostgres/current/bin/vacuumdb -d VCDB -e -v -f -U postgres > /tmp/vacuumdb.log
  • OUTPUT : 
    • vacuumdb: error: processing of database "VCDB" failed: ERROR: could not extend file "pg_tblspc/16397/PG_14_202107181/16400/9146569": No space left on device
• On running this query from VCDB to determine the source of events :
  • SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;
  • OUTPUT :
    • 20116558 | com.vmware.vc.guestOperations.GuestOperation | VSPHERE.LOCAL \Administrator
• In vCenter server directory /storage/seat/vpostgres/eventtblsp  or tasktblsp , observed numerous event files which are significantly old.
• The following events were observed in /var/log/vmware/vpxd/vpxd.log on the vCenter server :
  • YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] Execute result code: -1
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] SQL execution failed: insert into VPX_EVENT_8 (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME,USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME,CHANGE_TAG_ID) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] Execution elapsed time: 2 ms
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] Statement diagnostic data from driver is 53100:0:1:ERROR: could not extend file "pg_tblspc/16397/PG_14_202107181/16400/6662762": No space left on device;
--> Error while executing the query
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] Bind parameters:
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] [0]datatype: 1, size: 4, arraySize: 0
YYYY-MM-DDThh-mm-ss.647-04:00 error vpxd[06552] [Originator@6876 sub=Default opID=EventManagerProcessJobs-6b553e89] [VdbStatement] value = 29412800

Environment

• vCenter Server Appliance 8.x
• vCenter Server Appliance 7.x

Cause

• This behavior occurs by design. When the available disk space is low, PostgreSQL (vPostgres) prevents the creation or extension of data files to avoid potential database corruption
• vCenter services fail to start as they cannot log or process new event entries in the vPostgres database due to space constraints.
• The vacuumdb and TRUNCATE commands also fail in this condition, since both operations require additional temporary disk space to rewrite and clean database values before committing changes.

Resolution

1. Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0
   • Power Off the vCenter Server VM. If the VM is inaccessible, then do this from ESXi host level.
   • Right-click the VM.
   • Click Edit Settings.
   • Select Virtual Disk.
   • Increase the size of the disk. Ensuring there is no snapshot attached to VM. 
   • Power On the vCenter VM.
2. After the virtual disk is increased, return to the SSH session and run this command to automatically expand any logical volumes for which the physical volumes are increased:
   
   • /usr/lib/applmgmt/support/scripts/autogrow.sh
3. Delete old tasks, events and statistics data in vCenter Server 5.x, 6.x, 7.x and 8.x
   • Download the script attached to the above KB "2110031_Postgres_task_event_stat_reset_event_sequence.sql" and place it in the /tmp directory of the vCenter.
   • Stop the vCenter Server services :
     • service-control --stop --all
   • Run this command to execute the script for 7.0 and later: 
     • /opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=<days to keep> -v EventMaxAgeInDays=<days to keep> -v StatMaxAgeInDays=<days to keep> -d VCDB -t -q -f <download-path>/2110031_Postgres_task_event_stat_reset_event_sequence.sql
       Note: After execution of the script event id sequence will be reset. Event ids for new events will start from 1. 
   • Start the vCenter Server services :
     • service-control --start --all