"error msg="backup failed" controller=backup error="error checking if backup already exists in object storage" error after enabling data protection backup creation fails in Tanzu Mission Control

Products

• Tanzu Mission Control

Issue/Introduction

This article explains how to configure the ‘customer provisioned’ S3 bucket.

Symptoms:

The following error is encountered when you try to create a backup in Tanzu Mission Control (TMC). The error can be viewed in the velero-###-x### pod running in the velero namespace.
 

error msg="backup failed" controller=backup error="error checking if backup already exists in object storage: rpc error: code = Unknown desc = RequestError: send request failed\caused by: .....x509: certificate signed by unknown authority" error.file="/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:547" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupController).runBackup" key=velero/test-cluster logSource="pkg/controller/backup_controller.go:273"

Cause

User may be using on-prem storage solution, such as MinIO or Scality, which uses self signed certificates. All pods are running fine but you observe the error "x509: certificate signed by unknown authority" for volume backup and the backups fail.

Resolution

This is a known issue and currently, only the CA Cert from a cluster's proxy configuration is added to the cert store. However, configuration for CA certs for each data protection for the target location is not yet supported. 

Workaround:

1. Check for the valid backup storage target locations in TMC. In addition, make sure there should be accessibility between your cluster and your target location.

   kubectl get backupstoragelocations.velero.io -n velero

   NAME AGE

   sample-target-location 5d1h

2. First do one of the following:

   • Provide the caCert: “self signed cert of s3 storage location in base64 encoded”
   • Set insecureSkipTLSVerify: “true” in the backup location configuration file
   Then restart the velero-### pod.

   kubectl edit backupstoragelocations.velero.io  <your-target-location> -n velero running under “velero” namespace.
   
   For instance, the following is a sample file:

   spec:

     config:

       bucket: aws-s3

       resourceGroup: dp-backup

       storageAccount: ###########

       insecureSkipTLSVerify: "true" 

       subscriptionId: ######-####-###-###

     objectStorage:

       bucket: aws-s3

       caCert:  //self signed cert of s3 storage location in base64 encoded format