Velero snapshot backup fails with error code 3014

Products

• VMware Tanzu Kubernetes Grid Management
• Tanzu Kubernetes Grid
• Tanzu Kubernetes Runtime
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid Plus
• VMware Tanzu Kubernetes Grid Plus 1.x

Issue/Introduction

Velero installed on TKGm cluster with Velero plugin for vSphere.

The backup completes with warnings

velero backup get
NAME     STATUS      ERRORS   WARNINGS             CREATED                         EXPIRES         STORAGE LOCATION   SELECTOR
bkup1   Completed    0        1                    2024-09-06 09:50:39 +0000 UTC   29d             velero-backups-location     <none>

 

Velero pod log shows a warning while taking taking the snapshot

time="2024-09-06T09:50:44Z" level=warning msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/bkup1 logSource="pkg/backup/item_backupper.go:600" name=pvc-#################### namespace= persistentVolume=pvc-################# resource=persistentvolumes volumeSnapshotLocation=velero-backups-location

 

DataMgr Pod log shows that it cant copy the snapshot to backup repository

time="2024-09-06T09:50:51Z" level=info msg="CopyToRepo Error Received: Open virtual disk file failed. The error code is 3014. with error code: 3014" controller=upload generation=1 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:369" name=upload-################## namespace=velero phase=New snapshotID="ivd:########################"

Environment

TKGm with Velero and Velero vSphere plugin

 

Cause

The Velero vSphere user does not have the necessary permissions to take the snapshot.

The VDDK logs on the DataMgr pod show insufficient permissions error message

kubectl cp <DataMgr Pod>:/tmp/vmware-root/vixDiskLib-*.log <local path>

Error 3014 (Insufficient permissions in the host operating system) (No permission to perform this action.)

 

A secret was created in the velero namespace at install time and this will have details of the Velero vSphere user

kubectl get secret -n velero

Resolution

Review the Velero plugin for vSphere prerequisites 

Update the user to have the necessary credentials and privileges for VMDK Access

Additional Information

12/10/2024 - Updated 'Broken link' for VMware docs to BC techdocs