Velero backup-driver pods are not running

Products

• VMware vSphere with Tanzu

Issue/Introduction

• There are two pods for backup-driver:

# kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
backup-driver-xxxxxxxxxx-xxxxx 0/1 Pending 0 33m
backup-driver-xxxxxxxxxx-xxxxx 1/1 Running 0 42m
velero-xxxxxxxxxx-xx.xxx 1/1 Running 0 148m

 

• Describing Pending pod (backup-driver-6c758ff9cf-6ntrg) there was this warning:
  • k describe pod backup-driver-6c758ff9cf-6ntrg -n namespace_name

Warning FailedScheduling 7s (x10 over 6m48s) default-scheduler 0/10 nodes are available: 1 node had untolerated taint (node.kubernetes.io/unreachable: }, 3 nodes(s) didn't have free ports for the requested pod ports, 6 node(s) didn't match Pod's node affinity/selector. preemption: 0/10 nodes are available: 3 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling.

 

• When trying to take a backup, you get this error:

# velero backup create ako-system --namespace=vmware-system-ako
An error occurred: backups.velero.io is forbidden: User "sso:"[email protected]" cannot create resource "backups" in API group "velero.io" in the namespace "vmware-system-ako"

# velero backup create ako-system --namespace=default
An error occurred: backups.velero.io is forbidden: User "sso:"[email protected]" cannot create resource "backups" in API group "velero.io" in the namespace "default"

Resolution

• The number of replicas for the backup-driver deployment should be 1.
• As per the above symptoms,  there is one backup-driver working. We can delete the other one with the following command :
  • # kubectl delete pod <pod_name> -n <namespace_name>
• Velero backup error is caused by permissions error. Velero is not designed to take system namespaces. Velero is supposed to be used to backup user namespaces where applications are running. Default namespace or vmware-system-ako are system namespaces. 
• Check https://velero.io/docs/main/rbac/ and make sure the service account used to run backups has permissions to take backups.

Additional Information

Install and Configure the Velero Plugin for vSphere on a Supervisor