BOSH fails with Unable to render jobs for instance group. Expected variable to be already versioned in deployment

Products

• Operations Manager

Issue/Introduction

An upgrade of a deployment was performed and failed.

BOSH recreate / cck was performed in order to address upgrade failures.

BOSH encounters error such as:

Unable to render jobs for instance group 'apply-addons'.
Errors are: Unable to render templates for job 'bosh-dns'. 
Errors are: Expected variable '/dns_api_tls_ca' to be already versioned in deployment 'service-instance_5c166475-9b84-4192-92d9-c9512c441089'

Environment

Product Version: 2.6

Resolution

The solution to variable errors such as above is to run bosh deploy rather than recreate or cck to address unresponsive or missing VMs. If the failure occurs for an on-demand service-instance, then the solution is to re-run upgrade-all-errand which will trigger deploy for service instances.

This problem occurs if deployment is upgraded and failed and operator attempts to fix issue with recreate/cck. This error occurs because upgraded code may expect variables in credhub that have not been updated yet because deploy has not gotten to that point yet.

An example of steps that may lead to this state are:

1. Operator upgrades a deployment from 1.x to 2.x
2. Upgrade fails due to some job not starting on upgraded VM.
3. Operator attempts to recreate / delete + cck failed VM.
4. Operator then encounters error with unexpected variables.

This may lead to operator attempting to troubleshoot bosh unexpected variable error which is a misguided approach. In the above scenario, the operator needs to re-deploy and troubleshoot why the job failed after upgrade rather than bosh variable error.

If assistance is needed in determining why a job failed during upgrade then please open a ticket with Support.