BOSH deploy reports "Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch"

Products

• Operations Manager

Issue/Introduction

Symptoms:

When a customer was trying to update one of the etcd partition jobs through BOSH, the job fails with the following error message:

Started updating job etcd_server-partition > etcd_server-partition/1 (b9799760-5b4f-4fdb-baee-2016c0380a02) (canary). Failed: Action Failed get_task: Task f5c7c48b-ae7c-4883-5032-7f77005ba5cd result: Applying: Applying job etcd: Applying package etcd-common for job etcd: Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch. Expected 594e2aa92ccf2188475b5e417de29e19d1e98d38, got 112ed609bab4fa5b45b108cc848ae7858084ca02 for blob /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get489845863 (00:00:01)


Error 450001: Action Failed get_task: Task f5c7c48b-ae7c-4883-5032-7f77005ba5cd result: Applying: Applying job etcd: Applying package etcd-common for job etcd: Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch. Expected 594e2aa92ccf2188475b5e417de29e19d1e98d38, got 112ed609bab4fa5b45b108cc848ae7858084ca02 for blob /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get489845863

Environment

Cause

One possible cause for this issue is a bug in BOSH that allowed agents to consider packages as successfully compiled even though the package upload failed. The Director ends up saving that SHA1 and considers that package successfully compiled. This is not the only cause, though. Anything that would cause the SHA1 that is saved in the director's database to not match the actual SHA1 of the file in the blob store would cause the same error.

 

Resolution

One way to fix this is to remove compiled packages and kick off deploy again.

These are the detailed steps:

1. Connect via Single SHell (SSH) into the director VM. 
2. sudo su.
3. /var/vcap/jobs/director/bin/director_ctl console.
4. Bosh::Director::Models::CompiledPackage.all.map(&:blobstore_id).
5. Bosh::Director::Models::CompiledPackage.all.each(&:delete). 6. Kick off OpsMgr deploy.

Step 4 shows which blobs will be deleted. Record the result, if in future we want to clean up blobs that will be deleted from the database.

Step 5 deletes all compiled packages in the system which mean that the Ops Manager deploy will try to recompile every single package.