Velero Repo Maintenance Job pod fails with "Fatal: config or key <key-value> is damaged: ciphertext verification failed"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Velero repo-maintenance-job pods are failing on the VKS cluster due to errors encountered during Restic prune operations. Specifically, the prune step fails with a ciphertext verification error, which prevents repository cleanup and may impact backup efficiency or retention.

$ k logs <velero-maintenance-job-pod> -n velero
time="2025-07-02T08:06:01Z" level=error msg="Restic command fail with ExitCode: 1. Process ID is 18, Exit error is: exit status 1" logSource="pkg/util/exec/exec.go:66"
time="2025-07-02T08:06:01Z" level=error msg="An error occurred when running repo prune" error="failed to prune repo: error running command=restic prune --repo=<REDACTED_REPO_URL> --password-file=<REDACTED_PASSWORD_PATH> --cache-dir=<REDACTED_CACHE_DIR>, stdout=, stderr=Fatal: config or key <REDACTED_KEY> is damaged: ciphertext verification failed\n: exit status 1" error.file="pkg/repository/restic/repository.go:123" error.function="(*RepositoryService).exec" logSource="pkg/cmd/cli/repomaintenance/maintenance.go:72"

Environment

VMware vSphere Kubernetes Service

Velero 1.12+

Cause

Velero’s Restic based backup jobs are failing due to corruption in the repository’s encryption data. This causes Restic to reject prune and unlock operations with ciphertext verification failed errors. Since multiple repositories show the same issue, it likely stems from transient I/O interruptions or degraded storage performance during backup writes.

Resolution

• If the problem happens while using Restic based backup, please Migrate to Kopia which replaces Restic as the default volume backup engine in Velero v1.12+. Restic is officially deprecated in Velero v1.15.
• If corruption errors still persist after upgrading to v1.15+, raise a support case with Broadcom for further investigation.
• It is advisable to perform the previous backups, backuprequests and corrupt repos that had occurred from the previous backups. Refer to Additional Information in this KB to perfom those steps.
• Then to migrate to Kopia you can upgare to Velero v1.15 and above, for example for v1.16.0 version:

  kubectl set image daemonset/node-agent \
    node-agent=velero/velero:v1.16.0 \
    --namespace velero

  
  https://velero.io/docs/v1.16/upgrade-to-1.16/

Additional Information

There are usually due to either password mismatching or there is genuine corruption. For corruption condition, you can attempt below:

https://github.com/k8up-io/k8up/issues/835

However as it is depreceated from V1.15, it is much more recommended to migrate to Kopia. In that case where previous corrupt repos cannot be recovered, you can follow below to remove them.

Delete the velero backups first:
velero backup delete <name>

This will create a backup delete request to delete the objects from object storage. If they hang on/fail/issues

Identify and delete any stuck DeleteBackupRequest objects:
kubectl get deletebackuprequests -n velero
kubectl delete deletebackuprequest <request_name> -n velero

Force Delete the Backup CR:
Attempt standard Kubernetes deletion of the Backup object:
kubectl delete backup <backup_name> -n velero


Remove corrupt repos and jobs
kubectl delete backuprepository <corrupt_repo_name> -n velero
kubectl delete job -l velero.io/repository=<corrupt_repo_name> -n velero

Remove Finalizers (If Deletion Hangs):
If the previous command hangs indefinitely, the Velero controller is failing to finish finalizer. Open a new terminal session and patch the object to strip finalizers. Note: However this is a last resort action and exercise caution before proceeding with that.