vPostgres log rotation fails after updating to vCenter Server 8.0U3g

Products

• VMware vCenter Server 8.0

Issue/Introduction

• The /storage/log partition reaches 90-100% capacity.
• Logs stop rotating for the following logs /var/log/vmware/vpostgres/postgresql.log and /var/log/vmware/vtsdb/postgresql.log
• Disk exhaustion may lead to service instability.

Environment

VMware vCenter Server 8.0.x

 

Cause

• After upgrading vCenter to 8.0U3g, files vmware_postgres_log and vmware_vtsdb_postgres_log were merged into file /etc/logrotate.d/vmware-additional-data.lr
• Log rotation was not working as expected due to "postgresql.log" not being specified correctly.
• This leads to log files not being rotated and eventually filling up the /storage/log partition.

Resolution

Take a valid VAMI-based backup or offline snapshots of ALL vCenter/PSC nodes in the SSO domain before continuing. See VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Workaround:

1. Using vi or another editor create the following files:
2. Add the below entry /var/log/vmware/vpostgres/postgresql.log  in file /etc/logrotate.d/vmware-additional-data.lr.

   /var/log/vmware/vpostgres/postgresql.log
   /var/log/vmware/vpostgres/postgresql-%d.log
   {
       rotate 30
       firstaction
           dim=$(date -d "$(date +%Y-%m-01) +1 month -1 day" +%d)
           find /var/log/vmware/vpostgres/ -name "postgresql.log-*gz" -mtime +$dim -exec rm {} \;
       endscript
       daily
       maxage 30
       dateext
       dateyesterday
       dateformat -%Y-%m-%d_%H:%M:%S
       compress
       copytruncate
   }

3. Add /var/log/vmware/vtsdb/postgresql.log in file /etc/logrotate.d/vmware-additional-data.lr.

   /var/log/vmware/vtsdb/postgresql.log
   {
      rotate 30
      firstaction
         dim=$(date -d "$(date +%Y-%m-01) +1 month -1 day" +%d)
         find /var/log/vmware/vtsdb/ -name "postgresql.log-*gz" -mtime +$dim -exec rm {} \;
      endscript
      daily
      maxage 30
      dateext
      dateyesterday
      dateformat -%Y-%m-%d_%H:%M:%S
      compress
      copytruncate
   }

Note: No other action should be needed as the cron job will run every hour. 

 

Note: If "postgresql.log" has already grown to +20gb, The file must be zeroed out the file in order for log rotation to work. 

>/var/log/vmware/vpostgres/postgresql.log

or

sudo truncate -s 0 /storage/log/vmware/vpostgres/postgresql.log

Additional Information

• vCenter Server vPostgres log rotation fails after 30 days due to date format
• vCenter Server vPostgres log rotation fails due to missing vmware_postgres_log and vmware_vtsdb_postgres_log after updating to vCenter 8.0U3g (Japanese)
• Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0