velero - repo-maintainance-job job and pod resources piling up in velero namespace

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

After upgrading to Velero version v1.15.2_vmware.1, a large number of Kubernetes Job and Pod resources with the name pattern repo-maintain-job-* were observed being created in the velero namespace.

This is causing resource clutter and raised concerns regarding expected behavior and cleanup.

$ kubectl get job -n velero
NAME                              COMPLETIONS   DURATION   AGE
repo-maintain-job-1750552393648   1/1           7s         9d
repo-maintain-job-1750579086659   1/1           10s        8d
repo-maintain-job-1750579100368   1/1           14s        8d
repo-maintain-job-1750579124770   1/1           9s         8d
repo-maintain-job-1750579124825   1/1           10s        8d

$ kubectl get pods -n velero | grep -i repo-maintain
repo-maintain-job-1750552393648-mxx95   0/1     Completed   0          9d
repo-maintain-job-1750579086659-7cbqv   0/1     Completed   0          8d
repo-maintain-job-1750579100368-trzwq   0/1     Completed   0          8d
repo-maintain-job-1750579124770-kns77   0/1     Completed   0          8d

Environment

Velero1.15.x

Cause

Starting with Velero v1.15.2, the repository maintenance feature creates periodic jobs for backup repositories to perform maintenance operations.
By default, Velero keeps three completed jobs for each backup repository. This leads to a rapid increase in job and pod objects, especially in environments with multiple repositories or frequent backup cycles.

This behavior is documented here:
 Velero Repository Maintenance - Number of Jobs Retained

Resolution

To limit the number of retained jobs, the following configuration parameters were added to the Velero deployment  values.yaml under args section

    --default-repo-maintain-frequency=2m 
    --keep-latest-maintenance-jobs=3

After applying the changes, the number of repo-maintain-job-* resources was significantly reduced.