unable to restore backup of our existing cluster into new cluster "error restoring packages.data.packaging.carvel.dev/tkg-system/pinniped.tanzu.vmware.com: the server is currently unable to handle the request"
Article ID: 405269
Updated On:
Products
Issue/Introduction
Restore of the velero backup partially fails. This can be confirmed by describing the restore "velero restore describe <restore-name>"
You will be able to see errors on the describe of restore like below
could not restore, CustomResourceDefinition "backuprepositories.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.error restoring packages.data.packaging.carvel.dev/packages/cert-manager.tanzu.vmware.com.1.11.1+vmware.1-tkg.1: the server is currently unable to handle the request
Velero restore logs can be verified using velero restore logs <restore-name>time="2025-07-23T09:25:27Z" level=error msg="error restoring cert-manager.tanzu.vmware.com.1.1.0+vmware.1-tkg.2: the server is currently unable to handle the request" logSource="pkg/restore/restore.go:1475" restore=velero/test-backup-name
Environment
Velero v1.11.1_vmware.1
TKG 2.x
Cause
Velero’s restore describe confirms CRD-related warnings
could not restore, CustomResourceDefinition "backuprepositories.velero.io" already exists.Warning: the in-cluster version is different than the backed-up version.
Webhook Initialization Race Condition:PackageInstall and PackageMetadata CRs were restored while kapp-controller webhook was still initializing.
This resulted in HTTP 503 errors from the Kubernetes API.error restoring packages.data.packaging.carvel.dev/tkg-system/pinniped.tanzu.vmware.com.0.24.0+vmware.1-tkg.2: the server is currently unable to handle the request
Resolution
Carvel-managed packages (e.g., PackageInstall, PackageMetadata) should not be backed up or restored using Velero. Velero is primarily intended for backing up stateless workloads and application-level resources. It is not designed to handle VKS-managed infrastructure components that are orchestrated via in-cluster controllers.
In this case, restore failures were observed because Velero attempted to restore Carvel package CRs during webhook initialization. This led to API 503 responses, resulting in a PartiallyFailed restore phase. These failures stem from controller timing dependencies, not resource duplication.
To avoid such restore failures, we need to exclude VKS-managed resources from Velero operations by applying resource filtering. This allows backups and restores to focus strictly on workload-level objects. Reference: https://velero.io/docs/v1.16/resource-filtering/
Feedback
