Get "Error: Unable to render instance groups for deployment...Blobstore object 'GUID' not found" during BOSH deployment
Article ID: 298203
Updated On:
Products
Issue/Introduction
During a BOSH deployment, the task fails with following while rendering instance groups:
Task 12345 | 20:15:20 | Preparing deployment: Rendering templates (00:02:25) L Error: Unable to render instance groups for deployment. Errors are: Blobstore object '<BLOB_GUID>' not found Blobstore object '<BLOB_GUID>' not found Task 12345 | 20:17:45 | Error: Unable to render instance groups for deployment. Errors are: Blobstore object '<BLOG_GUID>' not found Updating deployment: Expected task '12345' to succeed but state is 'error' Exit code 1 Blobstore object '<BLOB_GUID>' not found
Environment
Cause
In BOSH deployment, jobs in the manifest depend on BOSH releases. If a release for a job is missing or corrupted, the deployment will fail with the error.
Resolution
To resolve this issue, we recommend the operator follow the steps below:
1. Collect and output to a file the failed task details with the debug flag enabled.
bosh task <FAILED_TASK_ID> --debug > task.debug.log --------------------------------------------------------------------------------------- # Example bosh task 12345 --debug > task.debug.log
2. Search the object ID (BLOG_GUID) that was displayed in the original error message.
3. One of the results from the search in the previous step will contain the job that is being downloaded:
root# cat task.debug.log| grep <BLOB_GUID> D, [2021-10-13T20:16:41.998014 #10438] [task:12345] DEBUG -- DirectorJobRunner: Downloading job 'metric_registrar_orchestrator' (BLOB_GUID)... Blobstore object ''<BLOB_GUID>'' not found
4. From the results of the previous step, record the job name associated with the missing Blobstore ID. In the example above, this would be job metric_registrar_orchestrator.
5. Download the manifest of the deployment (tile) that triggered this error with this command:
bosh -d <deployment name> manifest > manifest.yml --------------------------------------------------------------------------------------- # Example bosh -d cf-12345abcd manifest > manifest.yml
6. On the manifest.yml file, search for the failing job. For our example it would be metric_registrar_orchestrator. Identify what release(s) the job is relying on:
jobs:
- name: metric_registrar_orchestrator <---------
release: metric-registrar <-------
consumes: {}
provides: {}
properties:
.
.
.
7. Record the release name. For our example it would be metric-registrar. On the same manifest, under the releases section, identify the version of the release that should be used for this deployment.
releases:
- name: metric-registrar
version: 1.1.9
exported_from:
- os: ubuntu-xenial
version: '621.141'
url: file:///var/tempest/releases/metric-registrar-1.1.9-ubuntu-xenial-621.141.tgz
This is the release that is either missing or corrupted. To confirm the release status inspect it by running the command:
bosh inspect-release <Release Name>/<Release Version> --------------------------------------------------------------------------------------- # Example bosh inspect-release metric-registrar/1.1.9
The result for the command above will ether be:
- A 404 Not Found error if the release is actually missing.
- Or it will display the job and Blobstore ID that is missing where the release is most likely corrupt.
8. For either case, the release will need to be re-uploaded using --fix flag.
9. A copy of the release should be already on Operations Manager VM under /var/tempest/releases or you can download the product from Customer Support Portal, and unarchive it.
10. `bosh upload-release` with --fix to replace corrupted release:
bosh upload-release <release name> --fix --------------------------------------------------------------------------------------- # Example bosh upload-release metric-registrar-1.1.9-ubuntu-xenial-621.141.tgz --fix
11. If needed, repeat the steps above for any other missing Blobstore IDs.
12. After you complete the steps above, retry Apply Changes.
Feedback
