Velero Restore Failure for PVC Due to Missing Backup Blobs

Products

• Tanzu Kubernetes Runtime
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

A Velero restore operation fails for a specific PersistentVolumeClaim (PVC), even though the backup has completed successfully.
During restore, Velero reports errors indicating that it cannot read data from the backup location. This typically manifests as a failure in the data path service or kopia restore process.

Velero restore failure with the following error:

restore error: failed to read data from remote file
unexpected content error: error getting cached content from blob <blob-ID>
unable to complete GetBlob(<blob-ID>) despite multiple retries: unexpected EOF

 

velero restore logs:

level=info msg="adding persistentvolumes /<PVC-ID> to ItemBlock" backup=velero/<backup-name> logSource="pkg/backup/backup.go:641"
level=info msg="Backing up item" backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:184" name=<PVC-ID> resource=persistentvolumes
level=info msg="Executing takePVSnapshot" backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:549" name=<PVC-ID> resource=persistentvolumes
level=info msg="performing snapshot action for pv <PVC-ID>" backup=velero/<backup-name> logSource="internal/volumehelper/volume_policy_helper.go:95"
level=info msg="Skipping snapshot of persistent volume <PVC-ID>, because it's handled by CSI plugin." backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:576"
level=info msg="Skipping item because it's already been backed up." backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:179" name=<PVC-ID> resource=persistentvolumes

 

Running kopia content verify shows missing blob references:

Listing blobs...Listed <number-of-blobs> blobs.Verifying all contents...ERROR error content <content-ID> depends on missing blob <blob-ID>ERROR error content <content-ID> depends on missing blob <blob-ID>Finished verifying <total-contents>, found <error-count> errors.ERROR encountered <error-count> errors

Running kopia snapshot verify confirms missing blobs during snapshot validation:

Listing blobs...
Listed <number-of-blobs> blobs.
Processed <object-count> objects (<size>).
ERROR error processing <snapshot-ID>: object <object-ID> is backed by missing blob <blob-ID>
Finished processing <object-count> objects (<size>).
ERROR encountered <error-count> errors

 

Environment

Tanzu Kubernetes Grid Management 
Velero

Cause

The root cause was identified as missing or corrupted backup blobs in the object store.
Velero restore attempts to read blob data associated with the PVC but fails due to missing content.
Kopia validation (snapshot verify and content verify) confirms that several objects were backed by missing blobs.
This inconsistency in metadata and blob availability leads to restore failures, despite backups appearing successful.

Resolution

To ensure data integrity and resolve the restore issue, the following steps were taken:

1. Change Velero Backup Location

Please configure a new backup storage location to avoid reliance on corrupted/missing blobs and modify the following command as required:

velero install --provider aws --bucket <new-bucket-name> --secret-file <path-to-credentials> --plugins <aws-plugin-image> --backup-location-config region=<region>,s3ForcePathStyle="true",s3Url=http://<object-store-endpoint>,publicUrl=http://<object-store-endpoint> --use-node-agent --uploader-type=kopia --features=EnableCSI

2. Perform Fresh Backup and Restore

Additional Information

Existing backups tied to the old location cannot be restored if blobs are missing.
Always validate backup integrity using kopia verification commands before attempting restore.
For further details on blob handling and versioning, refer to: https://docs.min.io/enterprise/aistor-object-store/administration/objects-and-versioning