BBR backup for the TAS deployment is failing with error "NoSuchKey: The specified key does not exist"

Products

• VMware Tanzu Application Service

Issue/Introduction

BBR backup on CF deployed failed with below error.

└─$ less logs-debug.txt

Failed to execute: error while copying blobs: copying failed in parallel: operation error S3: CopyObject, response status 404, ReqID: 56AE119E568098381, HostID: 1037a12cb4abef92bb0dcd7ed1d0fbbf2650ff61bd9504dfcce72c46a3fe8b7a, API error NoSuchKey: Requested key is missing.

Encountered error while running backup for job `s3-unversioned-blobstore-backup-restorer` on `backup_restore/9dab84e5-d74c-11ec-b43c-0242ac110002`: Failed execution: error in blob copy: parallel copy failure: S3 CopyObject returned 404 - NoSuchKey. Exit code: 1.

Environment

CF deployment
Minio as a tile

Cause

This is due to corruption of the blobstore cache of one of the application.

Resolution

We need to identify the path in which we are facing the issue. This can be indicted using the race log of the minio.

Search for the ReqID ( example ReqID: 56AE119E568098381) from the BBR error o/p. 

<Error>
  <Code>NoSuchKey</Code>
  <Message>Requested key not found.</Message>
  <Key>buildpack_cache/73/48/7348be2a-a614-43ed-b91a-3d4790e00263/ubuntu-stack</Key>
  <BucketName>nyc-prod1-droplets</BucketName>
  <Resource>/backup-nyc-prod1-droplets/2024_06_12_05_07_32/droplets/buildpack_cache/73/48/7348be2a-a614-43ed-b91a-3d4790e00263/ubuntu-stack</Resource>
  <RequestId>56AE119E568098381</RequestId>
  <HostId>1037a12cb4abef92bb0dcd7ed1d0fbbf2650ff61bd9504dfcce72c46a3fe8b7a</HostId>
</Error>

Select the GUID from the above o/p, which is 7348be2a-a614-43ed-b91a-3d4790e00263. Search this in the source bucket buildcache and verify the content.


Check the CF api version and env and check if single app build cache deletion is supported. For example cf api version is 3.191.0

https://v3-apidocs.cloudfoundry.org/version/3.191.0/index.html#clear-buildpack-cache-for-application

POST /v3/apps/:guid/actions/clear_buildpack_cache

 cf curl -X POST/v3/apps/7348be2a-a614-43ed-b91a-3d4790e00263/actions/clear_buildpack_cache 


If deletion of single app build cache is not supported then use the below command to delete all the app buildcache. For example CF api version 3.152.0 doesn't support single app buildcache deletion

https://v3-apidocs.cloudfoundry.org/version/3.152.0/#clear-buildpack-cache

POST /v3/admin/actions/clear_buildpack_cache

 CF curl -X POST /v3/admin/actions/clear_buildpack_cache


Alternaltivly, to clean up the buildpack cache, admin users can run cf curl -X DELETE /v2/blobstores/buildpack_cache. This empties the buildpack cache completely.

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/4-0/tpcf/cc-blobstore.html

When the buildcache for a single application is corrupt, even after performing the above operation, the buildcache will not be deleted. If this is the case, you should manually delete the build cache.

example to delete the folder /7348be2a-a614-43ed-b91a-3d4790e00263/cflinux 

Additional Information

Note: In case you have any questions regarding any of the steps described in the above knowledge base, please open a support case before taking any further action.