Velero Backup Failed or PartiallyFailed due to Missing VolumeSnapshotter Configuration

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Velero backups are failing in PartiallyFailed state.

When describing the failed backup using velero CLI, an error message similar to the following is present:

velero describe backup <backup name> -n <backup namespace> --details

level=error msg="Error getting volume snapshotter for volume snapshot location"

Restores that are created from these PartiallyFailed backups may be missing data for the restored volumes, or any noted volumes in the error may not be restored.

Environment

Tanzu Kubernetes Grid 2.5 or vSphere Supervisor

Velero

Cause

Volumesnapshotlocation must be configured correctly for velero backups to complete properly.

In higher versions of velero, more optional parameters can be specified, including the config section.

The config section's values can vary by provider used for the backup.

Check the velero documentation based on the environment's velero version in use on details on optional parameters:

Velero 1.15 VolumeSnapshotLocation Documentation

 

Resolution

The exact resolution will depend on the fields that are missing or misconfigured in the volumesnapshotlocation object.

See the below for an example error and resolution step:

level=error msg="Error getting volume snapshotter for volume snapshot location error: /rpc error: code = Unknown desc = missing region in aws configuration"

1. Connect to the affected cluster with the failing back-ups
   
   
2. Retrieve the YAML details of the volumesnapshotlocation object:
   

   kubectl get volumesnapshotlocation  -A
   
   kubectl get volumesnapshotlocation <volumesnapshotlocation name> -n <volumesnapshotlocation namespace> -o yaml

    

3. If the volumesnapshotlocation is set as provider: aws, the above error message indicates that a spec.config.region value needs to be specified:
   

   apiVersion: velero.io/v1
   kind: VolumeSnapshotLocation
   metadata:
     name: <volumesnapshotlocation name>
     namespace: <volumesnapshotlocation namespace>
   spec:
     provider: aws

   In this scenario, the above volumesnapshotlocation is provider AWS which may require that the AWS region is specified.
   
   

4. Edit the volumesnapshotlocation to include the desired region configuration:
   

   apiVersion: velero.io/v1
   kind: VolumeSnapshotLocation
   metadata:
     name: <volumesnapshotlocation name>
     namespace: <volumesnapshotlocation namespace>
   spec:
     provider: aws
     config:
       region: <desired region>

    

5. Confirm that the next backup completes without the above error.