Troubleshooting manual cleanup failures and read-only states for vCenter /storage/archive partition

Products

• VMware vCenter Server

Issue/Introduction

The vCenter Server Appliance (VCSA) /storage/archive partition reaches 100% capacity and fails to clear even after executing the manual purge command find /storage/archive/vpostgres/ -mtime +300 | xargs rm.

Users may encounter errors such as "Operation not permitted" or "Read-only file system" when attempting to delete old WAL segments

Environment

• vCenter Server Appliance 7
• vCenter Server Appliance 8

Cause

• When the /storage/archive partition hits 100%, it can enter a read-only state to protect data integrity, preventing any manual or automatic deletion.
• The target cleanup directory may have incorrect ownership or may have been deleted during a previous troubleshooting attempt similar to vCenter VAMI patch failed on setting permissions for folder /storage/archive/vpostgres.

Resolution

1. Identify Read-Only State:
   1. Review /var/log/vmware/vpostgres/pg_archiver.log.stderr
      Look for "Read-only file system" within the log
      
      
   2. If the filesystem is read-only, reboot the VCSA to restore write operations, File system/storage/archive is not writable
      
      
2. Repair Directory & Permissions:
   1. Confirm the directory exists and set correct ownership with the following command from an SSH session to the vCenter: vCenter VAMI patch failed on setting permissions for folder /storage/archive/vpostgres
      cd /storage/archive/vpostgres
      
      
      
      
   2. Restore the permissions with the following command (the line will return empty):
      chown -R vpostgres:vpgmongrp /storage/archive/vpostgres
      
      
      
      
3. Perform Manual Clean up
   • Run the following command to delete files older than 90 days:
     find /storage/archive/vpostgres/ -mtime +90 | xargs rm
     
     
     Note: +90 represent 90 days, this can be adjusted to your specific needs. The default is +300
     
     
4. Restart vmware-postgres-archiver Service
   • Restart the service with the following command:
     service-control --stop vmware-postgres-archiver && service-control --start vmware-postgres-archiver
     
     

Additional Information

• The /storage/archive partition is designed to be full as part of standard vCenter operations. These steps are intended for resolving monitoring alerts or critical disk exhaustion. "File system /storage/archive is low on storage space" warning in vCenter Server Appliance 6.7/7.x/8.x
  
  
• For general information on this partition, refer to The full /Storage/archive partition causes an alert for the customer's monitoring application