Velero backups show Failed status due to velero pod "OOMKilled" crash

Products

• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x
• VMware Tanzu Mission Control
• VMware vSphere Kubernetes Service

Issue/Introduction

• TKG or VKS cluster backups fail with no logs and no errors
• Running velero backup describe <BACKUP_NAME> against the failed backup shows "Phase: Failed" with no events 
• Running velero backup logs <BACKUP_NAME> returns: "An error occurred: file not found"
• When the backup is run, the Velero pod and node-agents will crash and restart
• Describing the Velero pod with kubectl describe pod velero-<UNIQUE_POD_id> -n velero shows the pod running, but the following is shown:
  
  Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  
  
• Describing the node-agent pod with kubectl describe pod node-agent-<UNIQUE_POD_id> -n velero shows the pod running, but the following is shown: 
  

  Last State:     Terminated
    Reason:       OOMKilled
    Exit Code:    137

• Logging for the Velero pod reports no errors and appears to stop midway through backup operation
• With "apiVersion": "velero.io/v1", "kind": "Backup", Error: "failureReason": "found a backup with status "InProgress" during the server starting, mark it as "Failed". 
  • This failure means that the Velero pod was restarted in the middle of the backups.
  • This can happen if the Velero pod and the node-agents are being restarted due to OOMKilled.

Environment

TKG
TKGm
TKGS
VKS

Cause

This condition occurs because Velero pods and the node-agents have insufficient memory to satisfy the backup demands.
Memory limits on the Velero pods are reached, causing the vmkernel to OOMkill the pod or daemon set. 

Resolution

Modify the Velero deployment and node-agent daemonset to increase the memory limit.
See the example below for guidance:

kubectl edit deployment velero -n velero

Example:

    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  128Mi

Change to:

    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     500m
      memory:  128Mi

The limit will be workload dependent and may required more or less than what is suggested in the example. 
Monitor and test workloads to identify the necessary memory allocation for successful backup operations. 

NOTE: Modifying the memory limit will cause a re-rollout for the Velero deployment.

Alternative backup methods may also be utilized to reduce the impact of the high resource utilization of FSB. 
See the following documentation for Velero CSI-Snapshots:
Backup and Restore Using Velero with CSI Snapshot