Timeout during Velero backup - 30 minutes (+)

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Velero backups when described with below command are showing as PartiallyFailed and a time out of approximately 30 minutes.

velero backup describe my-backup --details

Environment

vCenter version and build: 8U3g
ESXi version and build: 8U3e
TKGs version: 3.3.0
Supervisor version v1.29.7

Cause

After the volume is created from the CSI snapshot, Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this DataUpload CR. The timeout is configurable through a node-agent’s parameter data-mover-prepare-timeout, the default value is 30 minutes. In this scenaro the backup with ~500 GB.

Resolution

Update the velero node-agent daemonset.apps to include the data-mover-prepare-timeout and set to a required setting.

In the below setting data-mover-prepare-timeout has been set to 3hours, the default is 30 minutes.

kubectl describe daemonset.apps/node-agent -n velero
Name:           node-agent
...
Pod Template:
  ...
  Containers:
   node-agent:
    ...
    Args:
      node-agent
      server
      --features=EnableCSI
      --data-mover-prepare-timeout=3h   # <-- Add this line to update the default of 30minutes to 3 hours -->
    ...

Additional Information

Details on the argument found on external velero link -> Velero -> https://velero.io/docs/main/csi-snapshot-data-movement/#:~:text=After%20the%20volume,is%2030%20minutes.