Not all Persistent Volumes (PVs) in a workload cluster are included in Velero backup.
search cancel

Not all Persistent Volumes (PVs) in a workload cluster are included in Velero backup.

book

Article ID: 414846

calendar_today

Updated On:

Products

VMware Tanzu Mission Control

Issue/Introduction

  • From Velero backup logs, following events are observed :

root@<tkgm-jumpbox>:~# velero backup logs <backup-name> |grep pvc-########-####-####-####-############
time="YYYY-MM-DDThh-mm-ss:45Z" level=info msg="Backing up item" backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:179" name=pvc-########-####-####-####-############ namespace= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:45Z" level=info msg="Executing takePVSnapshot" backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:509" name=pvc-########-####-####-####-############ namespace= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:45Z" level=info msg="Backup has volume snapshots disabled; skipping volume snapshot action." backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:512" name=pvc-########-####-####-####-############ namespace= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:58Z" level=info msg="Processing item" backup=velero/<backup-name> logSource="pkg/backup/backup.go:365" name=pvc-########-####-####-####-############ namespace= progress= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:58Z" level=info msg="Skipping item because it's already been backed up." backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:174" name=pvc-########-####-####-####-############ namespace= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:58Z" level=info msg="Backed up 103 items out of an estimated total of 24668 (estimate will change throughout the backup)" backup=velero/<backup-name> logSource="pkg/backup/backup.go:405" name=pvc-########-####-####-####-############ namespace= progress= resource=persistentvolumes
time="YYYY-MM-DDThh-mm-ss:05Z" level=info msg="Processing item" backup=velero/<backup-name> logSource="pkg/backup/backup.go:365" name=pvc-########-####-####-####-############ namespace=kube-system progress= resource=cnsvolumeoperationrequests.cns.vmware.com
time="YYYY-MM-DDThh-mm-ss:05Z" level=info msg="Excluding item because namespace is excluded" backup=velero/<backup-name> logSource="pkg/backup/item_backupper.go:140" name=pvc-########-####-####-####-############ namespace=kube-system resource=cnsvolumeoperationrequests.cns.vmware.com
time="YYYY-MM-DDThh-mm-ss:05Z" level=info msg="Backed up 1558 items out of an estimated total of 23597 (estimate will change throughout the backup)" backup=velero/<backup-name> logSource="pkg/backup/backup.go:405" name=pvc-########-####-####-####-############ namespace=kube-system progress= resource=cnsvolumeoperationrequests.cns.vmware.com

time="YYYY-MM-DDThh-mm-ss:15Z" level=info msg="Summary for skipped PVs: [

{\"name\":\"pvc-########-####-####-####-############\",\"reasons\":[{\"approach\":\"podvolume\",\"reason\":\"volume <volume-name> is declared in pod <pod-name> but not mounted by any container, skipping\"},{\"approach\":\"volumeSnapshot\",\"reason\":\"backup has volume snapshots disabled\"}]},

{\"name\":\"pvc-########-####-####-####-############\",\"reasons\":[{\"approach\":\"podvolume\",\"reason\":\"volume <volume-name> is declared in pod <pod-name> but not mounted by any container, skipping\"},{\"approach\":\"volumeSnapshot\",\"reason\":\"backup has volume snapshots disabled\"}]}

]" backup=velero/<backup-name> logSource="pkg/backup/backup.go:445"

  • On the TMC dashboard > Clusters > Data Protection, it shows FSB is enabled and CSI snapshot is also enabled but not ready as prerequisites in the cluster are not met.

  • Running the command "kubectl get volumesnapshotclass" returns an error stating that the server does not have a resource type “volumesnapshotclass”.
  • Checking for CRDs related to snapshots using "kubectl get crds | grep volumesnapshot" gives only the following output :
    • volumesnapshotlocations.velero.io 
  • The following components are missing from the above CRDs:
    • volumesnapshotclasses.snapshot.storage.k8s.io
    • volumesnapshotcontents.snapshot.storage.k8s.io
    • volumesnapshots.snapshot.storage.k8s.io
  • Verify if the snapshot controller is running using "kubectl get pods -n kube-system | grep snapshot-controller". If the output is empty, then the snapshot controller is not installed.

Environment

  • Tanzu Mission Control
  • Tanzu Kubernetes Grid Management 2.5.4
  • Kubernetes version 1.32

Cause

  • The backup was created with volume snapshots disabled, preventing Velero from taking CSI snapshots of PersistentVolumes.
  • The volume associated with the PVC was declared in the Pod specification but not mounted by any container, making it ineligible for pod-level volume backup.

Resolution

  1. Install the Kubernetes Volume Snapshot API components using the provided script in the below link. Refer to Volume Snapshot and Restore for installation steps.
  2. Create and correctly configure VolumeSnapshotClass. Refer to Volume Snapshot and Restore for installation steps.
  3. Recheck that snapshots are recognized in Tanzu Mission Control (TMC) by running "kubectl get volumesnapshotclass"
  4. Test a backup to ensure the snapshot feature is functioning as expected.

Additional Information

Powered by Wolken Software