Error: "BackupStorageLocation "bsl-aws" is unavailable...send request failed caused by...failed to verify certificate: x509: certificate signed by unknown authority" when attempting to Start Protection of a Kubernetes cluster

Products

• VMware Cloud Director

Issue/Introduction

• Attempting to Start Protection of a Kubernetes cluster using the Object Storage plugin in the Cloud Director Tenant portal fails.
• In Object Storage > Kubernetes Protection the Kubernetes cluster shows a Status of Failed and an error is displayed of the form:
  
  BackupStorageLocation "bsl-aws" is unavailable: rpc error: code = Unknown desc = RequestError: send request failed caused by: Get "https://<object_storage_endpoint>/<bucket_name>?delimiter=%2F&list-type=2&prefix=": tls: failed to verify certificate: x509: certificate signed by unknown authority
  
  
• Getting the backupstoragelocation with kubectl shows the PHASE as Unavailable in the Kubernetes cluster:
  
  kubectl get backupstoragelocation -n velero bsl-aws

NAME      PHASE         LAST VALIDATED   AGE    DEFAULT
bsl-aws   Unavailable   <invalid>        #      true
  
  
• Describing the backupstoragelocation with kubectl shows a Status: Message similar to the following:
  
  kubectl describe backupstoragelocation -n velero bsl-aws

Status:
  Last Validation Time:  <timestamp>
  Message:               BackupStorageLocation "bsl-aws" is unavailable: rpc error: code = Unknown desc = RequestError: send request failed
caused by: Get "https://<object_storage_endpoint>/<bucket_name>?delimiter=%2F&list-type=2&prefix=": tls: failed to verify certificate: x509: certificate signed by unknown authority

Environment

• VMware Cloud Director 10.6.x
• VMware Cloud Director Object Storage Extension 3.1.x

Cause

This issue occurs when SSL certificate of Object Storage Extension(OSE) is self-signed for the endpoint address.

To confirm the endpoint address of OSE, run the ose endpoint show OR sudo ose endpoint show command on the OSE server and view the OSE Endpoint URL value.

Resolution

To resolve the issue use a signed certificate with OSE's endpoint or set the optional oss.k8s.velero.insecure.skip.tls.verify parameter to true to skip the TLS verification for the OSE self-signed certificate from the Velero agent.

After making the changes return to the Cloud Director Tenant portal, Object Storage > Kubernetes Protection, and for the affected Kubernetes cluster choose All Actions > Stop Protection and then Start Protection again.

Additional Information

Before your tenants can start backing up and restoring clusters