Velero backup skips resources in Deleting state

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Velero backup operations for vSphere Kubernetes Service guest clusters finish with a "Completed" status, but specific resources such as Secrets or ConfigMaps are absent from the resulting backup bundle.

Environment

VMware vSphere Kubernetes Service 
Velero Standard Package for VKS 

Cause

Objects marked for deletion are skipped by Velero during backups.

Steps to verify :

1. Create a log bundle from the backup using command velero debug --backup <backup name> and extract the bundle.
2. Verify if the object was skipped due to deletion by looking for entry Skipping item because it's being deleted :

   time="YYYY-MM-DDT12:31:03Z" level=info msg="Processing item" backup=velero/<backup name> logSource="pkg/backup/backup.go:440" name=<object name> namespace=<object namespace> progress= resource=<object type>
   time="YYYY-MM-DDT12:31:03Z" level=info msg="Skipping item because it's being deleted." backup=velero/<backup name> logSource="pkg/backup/item_backupper.go:146"
   time="YYYY-MM-DDT12:31:03Z" level=info msg="Backed up ##  items out of an estimated total of ### (estimate will change throughout the backup)" backup=velero/<backup name> logSource="pkg/backup/backup.go:500" name=<object name> namespace=<object namespce> progress= resource=<object type>
   

3. Verify if the object has a deletionTimestamp from the object yaml : kubectl get <object> <object-name> -n <object-namespace> -oyaml

Resolution

If an object has been marked for deletion , make sure that the deletion is complete and then re-apply the object prior to the backup.

1. Verify if the object is stuck in the deleting state due to a finalizer from the object yaml.
2. Complete deletion of the object by addressing the finalizer and recreate the object if necessary.

Additional Information

Code reference :
velero-io/velero · GitHub

	if metadata.GetDeletionTimestamp() != nil {
		log.Info("Skipping item because it's being deleted.")
		return false
	}

 

Regarding deletionTimestamp :
When you tell Kubernetes to delete an object that has finalizers specified for it, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202 status code (HTTP "Accepted").
Finalizers | Kubernetes