S3 Storage Not Reclaimed During Kopia Garbage Collection Due to Incomplete Snapshots when using Velero
search cancel

S3 Storage Not Reclaimed During Kopia Garbage Collection Due to Incomplete Snapshots when using Velero

book

Article ID: 446231

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Kopia repository maintenance (Garbage Collection) runs successfully but fails to reclaim S3 backup storage. GC identifies large amounts of unused contents but only deletes a small fraction of unreferenced blobs. Continuous maintenance cycles complete without decreasing the overall S3 bucket size after 24+ hours.

Environment

VMware Kuberentes Service
Velero releases below v1.17

Cause

The Kopia backup repository contains stale and incomplete snapshots. This is a known issue with older versions of Velero and Kopia where incomplete snapshots are not automatically tracked or deleted by Velero. These incomplete checkpoints prevent blob rewrite and compaction from freeing space, as the data remains associated with orphaned, hidden checkpoints.

Resolution

Incomplete snapshots are untracked by Velero and must be manually deleted via the Kopia CLI to release S3 storage.

  1. Identify incomplete snapshots using the Kopia CLI. Note that most storage may be concentrated in specific large namespaces/PVCs rather than distributed evenly.

  2. Unpin the incomplete snapshot to make it eligible for deletion: kopia snapshot pin remove <SNAPSHOT_ID> --all (Example: kopia snapshot pin remove kc7279b25b3d7e71917aca3bfc4dbc976 --all)

  3. Delete the manifest record for the target snapshot: kopia snapshot delete <SNAPSHOT_ID> --delete

  4. Unlink the metadata pointers by running manual garbage collection: kopia snapshot gc

  5. Run a maintenance task with: kopia maintenance run --full  (Note: You may need to wait for several maintenance cycles for this to fully complete and retain free space. The aggresive method is with using the "--foce" option which may have drawbacks, hence not recommended for busy production workloads. For more information, refer to Additional Information)

  6. Perform a manual full maintenance on velero within the guest cluster(s). Note: This is required in older velero versions (v1.15 and lower) as newer velero versions perform these automatically:

    for repo in $(kubectl get backuprepositories -n velero -o jsonpath='{.items[*].metadata.name}'); do     echo "Forcing maintenance: $repo";     kubectl patch backuprepository "$repo" -n velero --type=merge -p '{                    
          "status": {
            "lastMaintenanceTime": "2024-01-01T00:00:00Z"
          }
        }'; done

Note: After snapshot deletion, multiple full maintenance runs must complete sequentially. The released data is marked for deletion by maintenance jobs but will not be physically purged from the S3 bucket until Kopia's internal safety time buffer expires.

Additional Information

 

  • Ensure you are only deleting untracked, incomplete snapshots. Incomplete snapshots are not used or tracked by Velero; unless manual data restoration via the Kopia CLI is required, they can be safely removed.

  • Review corresponding backup logs for underlying errors causing the incomplete checkpoints.

  • When validating small GC reclamations, confirm the overall repository footprint. Many repositories may inherently be small (e.g., kube-prometheus-stack), while the bulk of unreleased data resides in specific large volumes (e.g., integration-ms-ns).

  • Related GitHub tracking for older releases and memory increase issues:

 

Powered by Wolken Software