How to clean up Postgres .done file accumulation

Products

• VMware Tanzu Data Services
• VMware Tanzu for Postgres

Issue/Introduction

PostgreSQL only removes .done files and their corresponding 16MB WAL segments during checkpoints. If these files remain, it suggests PostgreSQL still requires the WAL files or the cleanup mechanism is not triggering.

Cause

There are instance where .done files become orphaned when there is a crash, backup script failure, or an OS glitch.

Resolution

1. Check WAL Retention Parameters

Please check the PostgreSQL WAL retention parameters. Even with healthy replication slots, Patroni environments often set these values high as a safety net to ensure a minimum number of WAL files remain on disk.

SHOW wal_keep_size;

NOTE: If wal_keep_size is set to something massive (e.g., 100GB), PostgreSQL will permanently retain that much WAL data, along with all of their .done files, before it starts deleting the oldest ones. Lower this value in your Patroni configuration if it is too high. 

 

2. Check for Lingering Replication Slots

Unused replication slots may be holding back the WAL. Please check for any inactive slots that should be removed.

SELECT slot_name, plugin, slot_type, active, restart_lsn 
FROM pg_replication_slots;

 

3. Confirm Checkpoints are completing. 

Verify that PostgreSQL checkpoints are completing successfully because failures or infrequent configurations prevent the removal of old WAL and .done files. Review your logs for checkpoint errors and check your checkpoint statistics to ensure proper cleanup.

 

4. Check for Orphaned Files 

A crash, backup script, or an OS glitch can cause .done files to become orphaned. This means the 16MB WAL file in pg_wal/ was deleted, but the 0-byte .done file in pg_wal/archive_status/ was left behind. Postgres will not clean up .done files if it doesn't see the corresponding WAL file during its checkpoint.

Compare the file counts. If you, have 2 million .done files in archive_status/ but only a few hundred 16MB WAL files in the parent pg_wal/ directory, this means you have orphaned files.

If the actual WAL files are gone, you can safely delete the orphaned .done files manually using find and rm commands, because PostgreSQL no longer cares about them.