PCF Installation fails due to failure fetching package blob from inner blobstore

Products

• Operations Manager

Issue/Introduction

Symptoms:
PCF Installation fail with error fetching blob:

Preparing package python-buildpack: Fetching package blob: Getting blob from inner blobstore: Getting blob from inner blobstore: Shelling out to bosh-blobstore-dav cli: Running command: 'bosh-blobstore-dav -c /var/vcap/bosh/etc/blobstore-dav.json get c24b8d55-9f46-49a9-9ef1-28838e54a615 /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get917676645', stdout: 'Error running app - Getting dav blob c24b8d55-9f46-49a9-9ef1-28838e54a615: Request failed, response: Response{ StatusCode: 404, Status: '404 Not Found' }', stderr: '': exit status 1

Environment

Cause

This problem is caused by BOSH Director referencing a blobID that does not exist or is corrupted on director blob store. For an internal blob store, the blobs will be stored on persistent disk of Director. This issue may occur as a result of corruption on that persistent disk such that blob data was lost. For external blob store, please contact your external blobstore provider to identify if there is any storage issues.

Resolution

The solution is to identify the corrupted release and force re-upload it with --fix option.

Perform the steps:
 

1. List out releases: bosh -e <env> releases
    
2. Inspect each release to dump blobID's associated with each release package: bosh -e <env> inspect-release <release name>/<version> 
    
3. Check the output of inspect-release for missing blobID mentioned in error ( c24b8d55-9f46-49a9-9ef1-28838e54a615 in example above). When you locate the release that contains the missing blobID, you have identified the release that needs fixing.
    
4. Re-upload the release with missing blobID: bosh -e <env> upload-release <release package.tgz> --fix
    
5. Retry apply changes in Ops Manager.