"context deadline exceeded" error observed when taking the velero backup on TKGM Cluster

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

When running a Velero backup for a Tanzu Kubernetes cluster, the backup status shows as "PartiallyFailed." The Velero logs or backup describe output display the following error:

rpc error: code = DeadlineExceeded desc = context deadline exceeded

This error occurs during the CSI snapshot preparation phase, indicating that the system timed out while waiting for a VolumeSnapshot to reach the "ReadyToUse" state.

Environment

TKGM 2.5.x

Cause

The Kopia CSI PVC plugin or the vSphere CSI controller encounters a timeout while preparing the data upload. The snapshot creation process fails to complete within the default timeout (typically 30 minutes for data movers or 4 minutes for CSI controller operations).

Resolution

Use the filesystem backup method, which bypasses the CSI snapshot timeout, or increase the timeout values if CSI snapshots are required.

Option 1: Use Filesystem Backup (Recommended)

• This method leverages direct filesystem operations instead of CSI snapshots. Initiate the backup using the --default-volumes-to-fs-backup flag:

velero backup create <BACKUP_NAME> --include-namespaces <NAMESPACE> --default-volumes-to-fs-backup --include-cluster-resources=true --wait

Option 2: Increase CSI Snapshot Timeouts

If CSI snapshots are required, increase the timeout limits for the node-agent and controller.

• Increase Data Mover Prepare Timeout: Edit the Velero node-agent DaemonSet:

kubectl edit ds node-agent -n velero

• • Add or update the data-mover-prepare-timeout argument:

spec:
  template:
    spec:
      containers:
      - name: node-agent
        args:
        - --data-mover-prepare-timeout=2h

• • Save and exit. The pods will restart automatically.

 

• Increase CSI Controller Timeouts (If necessary): Edit the vSphere CSI controller deployment:

kubectl edit deployment vsphere-csi-controller -n vmware-system-csi

Update the SNAPSHOT_TIMEOUT_MINUTES and PROVISION_TIMEOUT_MINUTES environment variables (e.g., increase from 4 to 10 minutes).