App fails during staging with missing blob for buildpack
search cancel

App fails during staging with missing blob for buildpack

book

Article ID: 297440

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

If a buildpack blob object has been removed from Blobstore without Cloud Controller being aware of the removal, pushing an application or restarting an application results in a staging error message "missing blob for xxx_buildpack". 
 

Staging app and tracing logs...
Staging error: Missing blob for staticfile_buildpack. Specify a different buildpack with the -b flag or contact your operator.
FAILED


The error is returned after the application contents have been uploaded.

######-MacBook-Pro:cf-sample-app-nodejs ########$ cf push -f manifest.yml
Pushing from manifest to org system / space identity-service-space as admin...
Using manifest file /Users/anthonyrios/Desktop/work/sample-apps/cf-sample-app-nodejs/manifest.yml
Getting app info...
Updating app with these attributes...
  name:                cf-nodejs
  path:                /Users/#######/Desktop/work/sample-apps/cf-sample-app-nodejs
  disk quota:          1G
  health check type:   port
  instances:           1
  memory:              512M
  stack:               cflinuxfs3
  routes:
    cf-nodejs-brash-panther-gn.cfapps-##.#####-##.###.#######.io

Updating app cf-nodejs...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...
 714.95 KiB / 714.95 KiB [==================================================================================================================] 100.00% 10s

Waiting for API to complete processing files...

Stopping app...

Staging app and tracing logs...
Staging error: Missing blob for staticfile_buildpack. Specify a different buildpack with the -b flag or contact your operator.
FAILED


Steps to validate removal of blob object

1. Obtain the GUID of the missing buildpack.

cf curl /v2/buildpacks | jq '.resources[] | .entity.name,.metadata.guid'


2. Access contents in TAS Blobstore. If using an internal Blobstore, you will need to SSH into the VM backing Blobstore. If using an external Blobstore ((eg Cloud Storage or Azure Storage) you can use their web explorer utility:

Internal Blobstore Instructions

  • SSH to NFS Server VM.  
    bosh -d cf-83f7743cacc5440eb145 ssh nfs_server/0
  • Change to buildpack directory. 
    cd /var/vcap/store/shared/cc-buildpacks
  • Cloud Controller uses the first 4 digits of the buildpack GUID to store the buildpack file. With first two digits used as the parent directory and 3rd & 4th digits as the child directory which contains the buildpack file.
  • For example if the staticfile_buildpack with GUID "ace103e6-ee61-4bcb-8dfb-c143e45e1816" existed in the blobstore then it would be stored in parent directory "ac" and child directory "e1". 
    nfs_server/547c30aa-17e5-436a-b887-388bf94234f3:/var/vcap/store/shared/cc-buildpacks# ls -l
total 52
drwx------ 3 vcap vcap 4096 Jul 14 21:18 04
drwx------ 3 vcap vcap 4096 Jul 14 21:18 0b
drwx------ 3 vcap vcap 4096 Jul 14 21:18 1f
drwx------ 3 vcap vcap 4096 Jul 14 21:17 3d
drwx------ 3 vcap vcap 4096 Jul 14 21:17 4e
drwx------ 3 vcap vcap 4096 Jul 14 21:18 53
drwx------ 3 vcap vcap 4096 Jul 14 21:18 55
drwx------ 3 vcap vcap 4096 Jul 14 21:17 7d
drwx------ 4 vcap vcap 4096 Jul 14 21:17 ac
drwx------ 3 vcap vcap 4096 Jul 14 21:18 d0
drwx------ 3 vcap vcap 4096 Jul 14 21:18 d6
drwx------ 3 vcap vcap 4096 Jul 14 21:18 f3
drwx------ 3 vcap vcap 4096 Jul 14 21:18 fa

nfs_server/547c30aa-17e5-436a-b887-388bf94234f3:/var/vcap/store/shared/cc-buildpacks# ls -l ac/
91/ e1/
nfs_server/547c30aa-17e5-436a-b887-388bf94234f3:/var/vcap/store/shared/cc-buildpacks# ls -l ac/e1
total 7844
-rw------- 1 vcap vcap 8030798 Jul 15 18:04 ace103e6-ee61-4bcb-8dfb-c143e45e1816_93a2286ac6f6d4394d8e591138168cb210c2f3f33bedbac3417af120bff64fb9

External Blobstore Instructions

  • Navigate to Web Explorer utility for your storage solution to access contents
  • Navigate to the directory for buildpacks. Note: the operator is responsible for naming the directory used to store buildpacks and is configured in the TAS tile. You can find the name of the buildpack name (PAS --> File Storage --> Configure your Cloud Controller's filesystem)
  • Cloud Controller uses the first 4 digits of the buildpack GUID to store the buildpack file. With first two digits used as the parent directory and 3rd & 4th digits as the child directory which contains the buildpack file

3. If the blob was removed from the Blobstore, you will not find the directories containing the buildpack file or the buildpack file itself.

Resolution

TAS Installed Buildpacks

If the buildpack removed is part of the TAS installation, in other words if the removed buildpack was bundled with the TAS tile, then you can re-upload the buildpacks by invoking the post-start script on cloud controller.

The post start script must be ran from the first Cloud Controller VM. This would be index 0.

1. SSH into Cloud Controller: 
bosh -d cf-83f7743cacc5440eb145 ssh cloud_controller/0

2. As root, execute the post-start script: 
sudo /var/vcap/jobs/cloud_controller_ng/bin/post-start


Third Party Managed Buildpacks

If the buildpack was installed through a third party tile (eg App Dynamics tile), then you will need to investigate if an errand is available to re-upload the buildpack for you. For exampl, App Dynamics will re-upload the appD buildpack if you run errand "deploy all".

Custom Buildpacks

If you uploaded a custom buildpack, then you will need to re-upload the buildpack yourself.

Powered by Wolken Software