vPostgres log rotation failure causing disk exhaustion in vCenter Server version 8.0U3g

Products

• VMware vCenter Server 8.0

Issue/Introduction

vCenter Server 8.0U3g environments encounter disk exhaustion alerts for /storage/log and /storage/archive. This issue occurs when vPostgres logs fail to rotate correctly, leading to massive growth of postgresql.log and the accumulation of unpurged WAL segments. Symptoms include critical alerts in the vCenter monitoring application and inability to perform system operations due to insufficient disk space.

Symptoms:

• /storage/log partition utilization exceeds 85%.
• /storage/archive partition utilization reaches critical thresholds (95%+).
• postgresql.log file size exceeds 1GB.

Environment

VMware vCenter Server 8.0U3g

Cause

A known defect in the vPostgres log rotation mechanism prevents the automated cleanup of logs and archive segments.

Resolution

This issue is resolved in vCenter server versions  8.0U3h or higher

 

To free up the space immediately and clear the alert, follow below steps:

1. Clear the large vPostgres log file: echo " " > /storage/log/vmware/vpostgres/postgresql.log
2. Remove WAL segments older than 60 days in the archive partition: find /storage/archive/vpostgres/ -type f -mtime +60 -exec rm {} \;
3. Truncate stdstream.log files if disk space remains critical: echo " " > /storage/log/vmware/wcp/stdstream.log-0.stderr
4. Restart the WCP service to release file handles: service-control --restart wcp

 

For a workaround, if upgrade is not feasible at this time, follow below steps:

Note: Take a valid VAMI-based backup or offline snapshots of ALL vCenter/PSC nodes in the SSO domain before continuing. See VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Workaround:

1. Using vi or another editor create the following files:
2. Add the below entry /var/log/vmware/vpostgres/postgresql.log  in file /etc/logrotate.d/vmware-additional-data.lr.

   /var/log/vmware/vpostgres/postgresql.log
   /var/log/vmware/vpostgres/postgresql-%d.log
   {
       rotate 30
       firstaction
           dim=$(date -d "$(date +%Y-%m-01) +1 month -1 day" +%d)
           find /var/log/vmware/vpostgres/ -name "postgresql.log-*gz" -mtime +$dim -exec rm {} \;
       endscript
       daily
       maxage 30
       dateext
       dateyesterday
       dateformat -%Y-%m-%d_%H:%M:%S
       compress
       copytruncate
   }

3. Add /var/log/vmware/vtsdb/postgresql.log in file /etc/logrotate.d/vmware-additional-data.lr.

   /var/log/vmware/vtsdb/postgresql.log
   {
      rotate 30
      firstaction
         dim=$(date -d "$(date +%Y-%m-01) +1 month -1 day" +%d)
         find /var/log/vmware/vtsdb/ -name "postgresql.log-*gz" -mtime +$dim -exec rm {} \;
      endscript
      daily
      maxage 30
      dateext
      dateyesterday
      dateformat -%Y-%m-%d_%H:%M:%S
      compress
      copytruncate
   }

Note: No other action should be needed as the cron job will run every hour. 

 

Note: If "postgresql.log" has already grown to +20gb, The file must be zeroed out the file in order for log rotation to work. 

>/var/log/vmware/vpostgres/postgresql.log

or

sudo truncate -s 0 /storage/log/vmware/vpostgres/postgresql.log

Additional Information

• vCenter log disk exhaustion or /storage/log full
• vCenter Server vPostgres log rotation fails after 30 days due to date format
• vCenter Server vPostgres log rotation fails due to missing vmware_postgres_log and vmware_vtsdb_postgres_log after updating to vCenter 8.0U3g (Japanese)
• Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0