Velero Restore fails for PVCs using WaitForFirstConsumer binding mode in VMware Telco Cloud Automation

Products

• VMware Telco Cloud Automation

Issue/Introduction

• When using the Velero addon in VMware Telco Cloud Automation to back up and restore Persistent Volume Claims (PVCs) on MinIO or other object storage, following behavior is observed:
  
  
• Backup job completes but the describe on the backup job returns below error:
  

  Warnings: Velero: message: /Explicitly setting empty volume-id to prevent snapshot operations. message: /Explicitly setting empty volume-id to prevent snapshot operations. Cluster: resource: /persistentvolumes name: /pvc-################ message: /No volume ID returned by volume snapshotter for persistent volume resource: /persistentvolumes name: /pvc-################ message: /No volume ID returned by volume snapshotter for persistent volume

• The restore of the PVC data fails with error shown below:
  

  error preparing persistentvolumeclaims/sample/test-####: rpc error: code = Unknown desc = Failed to create a CloneFromSnapshot CR: Phase=Failed, err=cloneFromSnapshot: Failed at calling SnapshotManager CreateVolumeFromSnapshotWithMetadata with peId pvc:sample/test-####, err: PVC "test-####" did not become Bound: PersistentVolumeClaims [test-####] not all in phase Bound within 3m0s Error creating volume from metadata 

Environment

TCA 3.2.0
TCP 5

Cause

This issue is caused by the configuration of the Velero addon within TCA, specifically regarding the Container Storage Interface (CSI) integration.

1. In the TCA Velero addon (versions 3.2 and 3.4), the EnableCSI parameter is set to false by default.
2. Because EnableCSI is false, Velero cannot use the native CSI snapshotter. It attempts to fall back to the legacy vSphere volume snapshotter. If that is not applicable, it sets an empty volume ID, generating the "No volume ID returned" warnings. These warnings are generally harmless if the backup operation completes successfully.
3. The velero-plugin-for-vsphere (without EnableCSI=true) does not support restoring PVCs associated with a StorageClass that uses the WaitForFirstConsumer VolumeBindingMode.
   1. When the vsphere-csi addon is deployed in TCA with Region and Zone configurations, it automatically creates a StorageClass with volumeBindingMode: WaitForFirstConsumer to ensure topology awareness.
   2. During restore, Velero attempts to provision the PVC, but without the CSI logic enabled, it cannot handle the delayed binding required by WaitForFirstConsumer, causing the restore to hang or fail.

Resolution

To successfully restore PVCs in the current TCA versions ensure the target StorageClass uses Immediate binding mode.

1. Modify the vSphere-CSI Addon Configuration:

   • Navigate to the TCA infrastructure settings where the vsphere-csi addon is managed.

   • Redeploy or Reconfigure the vsphere-csi addon.

   • Remove the Region and Zone configurations.

2. Verify StorageClass Mode:

   • Once the addon is redeployed without topology constraints, the resulting StorageClass will default to volumeBindingMode: Immediate.

3. Retry Backup/Restore:

   • Perform a new backup (if necessary to capture the new config) or retry the restore operation. Velero supports Immediate binding mode natively in this configuration.

Impact of Workaround: Redeploying the CSI addon does not impact existing PVs/PVCs or running applications. The CSI driver and StorageClass are primarily responsible for provisioning new volumes (such as those created during a restore). Once a PV is created, it is managed by the Kubernetes cluster controller.

Resolution

EnableCSI flag (true) in the Velero addon configuration in a future release of Telco Cloud Automation. This will allow full support for WaitForFirstConsumer binding modes and align TCA's Velero capabilities with the upstream standard.

Additional Information

Question: Is there a functional difference between the TCA Velero plugin and open-source Velero?
Answer: Functionally, the deployments operate similarly. However, the limitation described in this article is specific to the TCA Add-on configuration. While open-source Velero supports restoring PVCs with WaitForFirstConsumer binding mode (when configured correctly with CSI), the TCA add-on explicitly sets EnableCSI to false. This configuration choice prevents the plugin from utilizing the necessary CSI features to handle delayed binding during restores.

Question: Why is the EnableCSI parameter set to false in TCA 3.2 and 3.4?
Answer: This setting was retained during the uptake of TKG 2.5.4 components. In Velero version 1.15.2+vmware.1, the velero-plugin-for-csi was integrated into the velero-plugin-for-vsphere. However, the EnableCSI flag was not switched to true in the TCA add-on definition during this integration. This flag must be enabled to unlock full CSI backup/restore capabilities, and a fix is planned for a future release.

Question: Does TCA 3.4 fix this issue since it uses a newer Velero version (1.15.2)?
Answer: No, the limitation persists in TCA 3.4. Although TCA 3.4 includes Velero 1.15.2, the EnableCSI parameter remains false. Consequently, restoring PVCs with WaitForFirstConsumer binding mode is still unsupported.In TCA 3.2, the failure message was generic (RPC error... Unknown). In TCA 3.4, the system handles this condition explicitly and provides a clear error message: "The PVC to be restored is associated with StorageClass with WaitForFirstConsumer VolumeBindingMode, this is currently not supported.

Question: Why does removing Region/Zone from the CSI Addon fix the restore?
Answer: Removing the Region and Zone configurations forces the vsphere-csi driver to create the default StorageClass with Immediate VolumeBindingMode. The legacy vSphere volume snapshotter (used when EnableCSI is false) fully supports Immediate binding, allowing the restore to proceed successfully. This change only affects newly provisioned volumes (like those created during a restore) and does not disrupt existing bound volumes.