Velero restore operations failing with "Permission Denied" errors.

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Backup of resources were successfully created using Velero. However, the following errors were encountered during the restoration process.

Error log --

Not found: /restores/data/.velero/d898f20f-51d2-4a5b-a001-2050b430d7d7
Found /restores/data/.velero/d898f20f-51d2-4a5b-a001-2050b430d7d7All restic restores are done
unlinkat /restores/data/.velero/d898f20f-51d2-4a5b-a001-2050b430d7d7: permission denied

Environment

VMware Tanzu Kubernetes Grid 2.x

Cause

The restore operation failed because the backup was created by user ID 0 (root), while the restore was attempted by a normal userID 1001. This mismatch in user permissions resulted in the "Permission Denied" errors during the restore process because restore didn't have permissions to access the backup created by root user

Resolution

• To resolve the restore issue, the restore helper container was updated using the fs-restore-action-config ConfigMap to allow helper init containers to run with user ID 0. 
  
  • Docs -- https://velero.io/docs/v1.12/file-system-backup/#customize-restore-helper-container