Partially Failed Backups in Velero or Tanzu Mission Control (TMC) Data Protection due to BackupRepository error "repository not initialized in the provided storage"

Products

• VMware Tanzu Mission Control
• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

Velero backups are failing or showing as Partially Failed.

 

While connected to the workload cluster context where velero or TMC Data Protection is running, the following symptoms are observed:

• A describe of the latest failed backup shows an error message similar to the following:
  

  kubectl get backup -A
  kubectl describe backup -n <backup namespace> <latest backup name>
  
  Resource List: <error getting backup resource list

• The associated BackupStorageLocation (BSL) is healthy:

  kubectl get bsl -A

• Performing a describe on one or more associated backuprepository shows one of the error messages below:
  

  kubectl get backuprepository -A
  kubectl describe backuprepository -n <backuprepository namespace> <backuprepository name>
  
  "error to connect backup repo: error to connect repo with storage: error to connect to repository: repository not initialized in the provided storage"
  
  "BackupStorageLocation <backupstoragelocation name> not found"

• If included in the failed backup, one or more podvolumebackups are in Failed state:
  

  kubectl get podvolumebackup -A

Environment

vSphere Supervisor

Velero 1.13 and higher.

This issue can occur when using velero through Tanzu Mission Control Data Protection.

Cause

The velero backuprepository is out of sync with the underlying storage, such as minio.

Previously used backuprepository objects which are pointed to stale or now non-existent backupstoragelocation (BSL) or repositories in the underlying storage do not automatically update.

Velero is not aware of direct changes made manually in the underlying storage which can lead to mismatch with and data loss from the backuprepository object dedicated to the changed or deleted repository.

Velero does not save plain payload data directly to the underlying storage. It uses backup repository objects which support methods such as deduplication, compression and encryption to save the processed data to the underlying storage.

Unwanted backups should be cleaned up through velero CLI broadcom package or through the TMC Data Protection web UI.

For more information on velero CLI backup cleanup, see Velero Documentation on Backup Deletion

 

Resolution

These stale backuprepository objects will need to be cleaned up manually. Velero will automatically create a backuprepository if it does not exist once the corresponding backup is started.

The below steps will advise on locating backuprepository objects that are stale or disconnected which can be cleaned up safely.

1. Connect into the workload cluster context where velero is running, or where TMC Data Protection is running
2. Retrieve a list of all backuprepository objects in the affected workload cluster:
   

   kubectl get backuprepository -A

3. Perform a describe on the backupRepository to check if it has an error reaching the associated backupStorageLocation (BSL) or repository in the underlying storage:
   

   kubectl describe backuprepository -n <backuprepository namespace> <backuprepository name>
   
   "error to connect backup repo: error to connect repo with storage: error to connect to repository: repository not initialized in the provided storage"
   
   "BackupStorageLocation <backupstoragelocation name> not found"

4. If the repository noted in the above error message does not exist in the underlying storage (ex. minio), or the above backupStorageLocation (BSL) is not present in the environment, this backupRepository object can be cleaned up as it is considered stale:
   

   kubectl get bsl -A | grep <backupStorageLocation name from error>
   
   kubectl get backuprepository -n <backuprepostiory namespace> <backuprepository name>
   
   kubectl delete backuprepository -n <backuprepostiory namespace> <backuprepository name>

   Velero will automatically create a backuprepository if it does not exist once the corresponding backup is started.

5. Once the corresponding stale backupRepository object has been cleaned up, a backup can be initiated again and the new backupRepository objects will be automatically created.
   These can be checked if there are still repository connection errors:
   

   kubectl get backuprepository -A
   
   kubectl describe backuprepository -n <backuprepository namespace> <backuprepository name>

Additional Information

Velero CLI should be installed using the equivalent Broadcom Package and should ideally match the version matrix as per below:

Version Matrix for Velero