When it comes to understanding resource deletion in Kubernetes, knowledge of how finalizers work is helpful and can help you understand why some objects don’t get deleted.
Finalizers are keys on resources that signal pre-delete operations. They control the garbage collection on resources, and are designed to alert controllers what cleanup operations to perform prior to removing a resource. However, they don’t necessarily name code that should be executed; finalizers on resources are basically just lists of keys much like annotations. Like annotations, they can be manipulated.
Here’s what may happen upon attempting to delete a kubernetes object:
^ packagerepositories.packaging.carvel.dev CustomResourceDefinition 1y delete - delete ongoing Waiting on finalizers:
customresourcecleanup.apiextensions.k8s.io
Delete failed: Deleting: Error (see .status.usefulErrorMessage for details)
The deletion fails because it's waiting on the finalizer and remains stuck in the process of deletion. When we attempt to get that object again, we discover the object has been modified to include the deletion timestamp.
│ Metadata: │
│ Creation Timestamp: 2023-11-13T22:17:00Z │
│ Deletion Grace Period Seconds: 0 │
│ Deletion Timestamp: 2024-07-15T19:14:47Z │
│ Finalizers: │
│ finalizers.packagerepository.packaging.carvel.dev/delete
In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and blocked removal of the object from etcd. The deletion timestamp signals that deletion was requested, but the deletion will not be complete until we edit the object and remove the finalizer.
We can use the patch
command to remove finalizers. If we want to delete an object, we can simply patch it on the command line to remove the finalizers. In this way, the deletion that was running in the background will complete and the object will be deleted.
kubectl patch <object> -p '{"metadata":{"finalizers":[]}}' --type=merge