Valkey Service Instance Upgrades Fail with 404 for adbr_agent_cert
Article ID: 441298
Updated On:
Products
Issue/Introduction
During an upgrade to VMware Tanzu for Valkey 10.4.0, upgrading existing service instances fails when running the upgrade-all-service-instances errand or executing cf upgrade-service.
The associated BOSH task fails during template rendering with an error similar to:
Reference Error logs:
Error: Unable to render instance groups for deployment. Errors are:
- Unable to render jobs for instance group 'redis-instance'. Errors are:
- Unable to render templates for job 'adbr-agent'. Errors are:
- Failed to find variable '/p-bosh/service-instance_<GUID>/adbr_agent_cert' from config server: HTTP Code '404', Error: 'The request could not be completed because the credential does not exist...'
Newly created service instances on version 10.4.0 deploy successfully.
Environment
VMware Tanzu for Valkey 10.4.0
Cause
This issue is caused by a defect in the Valkey 10.4.0 tile migration logic.
Version 10.4.0 introduces the Automated Data Backup and Restore (ADBR) feature, which adds a required adbr-agent job to the service instance deployment manifest. This job depends on the adbr_agent_cert credential.
For preexisting service instances that:
- Were created on older tile versions, and
- Use non-TLS configurations,
The migration process fails to register the credential generation workflow for adbr_agent_cert.
As a result:
- BOSH attempts to render the adbr-agent job,
- Queries CredHub for /p-bosh/service-instance_<GUID>/adbr_agent_cert,
- Receives a 404 (credential not found),
- Fails the deployment.
Resolution
This issue will be fixed in a future release of VMware Tanzu for Valkey 10.4.x.
Subscribe to this article for updates on availability of the fix.
Workaround
To unblock upgrades, manually create the missing credential in CredHub.
The credential must:
- Be generated per affected service instance (<GUID>),
- Use production-grade parameters,
- Be signed by the platform Services CA (/services/tls_ca).
After creating the credential, rerun the upgrade operation.
Additional Information
Feedback
