Error: "volumesnapshots.snapshot.storage.k8s.io "<####################>" not found" while migrating workloads from TKG to VKS cluster.

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Workload migration from VMware Tanzu Kubernetes Grid Multicloud (TKGm) to vSphere with Tanzu (VKS) using standard Velero CSI backups fails during the restoration phase.
• Persistent Volume Claims (PVCs) remain in a "Pending" state on the destination cluster.
• The VKS CSI provisioner logs display the following errors:

failed to provision volume with StorageClass "<#############>": error getting handle for DataSource Type VolumeSnapshot by Name <#############>: snapshot <#############> is not Ready
Error: volumesnapshots.snapshot.storage.k8s.io "<#############>" not found on namespace: <#############> from Supervisor Cluster.

• The source TKGm workload cluster and the target workload cluster VKS are in different vCenter Servers.

Environment

TKGm 3.4+v1.33
Velero 1.12 to 1.16

Cause

• vSphere Container Storage Interface (CSI) First Class Disk (FCD) snapshots are tightly bound to the source vCenter Server Database (VCDB) and storage array.
• When restoring to a disparate vCenter Server, the destination VKS CSI driver cannot locate the source FCD snapshot ID, resulting in the snapshot not found error.

Resolution

Change the backup methodology on the source TKGm cluster from CSI snapshot-based backups to file-system backups to bypass the CSI plugin limitations:

velero backup create <REDACTED_SECRETS> --default-volumes-to-fs-backup