Error "daemonset pod not found in running state in node <control_plane_node>" during a guest cluster backup using Velero

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Running velero backup get shows a status of PartiallyFailed:
  
   NAME             STATUS            ERRORS    WARNINGS    CREATED           EXPIRES              STORAGE LOCATION       
<backup_name>   PartiallyFailed       #         #        <date/time>       <no._of_days>         <backup_location>       
              
• Describing one of the failed backups show below error:
  
  velero backup describe <backup_name>

name: /<pod_name> message: /Error backing up item error: /daemonset pod not found in running state in node <control_plane_node_name>

Environment

vSphere Kubernetes Service

Cause

By default, the Velero node-agent daemonset is only installed on Kubernetes worker nodes. If a pod is scheduled on a control plane node, the lack of a local node-agent prevents Velero from executing the backup for that pod.

Resolution

Choose one of the following resolutions based on your operational requirements:

• Reschedule workloads to worker nodes (Recommended):
  
  
  • As a Kubernetes best practice, custom workloads should not be deployed on control plane nodes.
  • Review the deployment or daemonset configuration.
  • Adjust tolerations and nodeSelector to ensure pods are scheduled only on worker nodes.
    
    
• Exclude the impacted namespace or objects:
  
  
  • If it is not necessary to back up the pods running on control plane nodes (such as monitoring agents), exclude them from the backup:
    
    velero backup create <backup-name> --exclude-namespaces <namespace-name>


• Skip specific pods using label selectors:
  
  
  • Apply the label backup-status:exclude to the pod or namespace.
  • If using Tanzu Mission Control (TMC), go to Cluster > Data Protection > Create backup and use the label selector to exclude this criteria.