Staging app failure with "Failed to upload payload for build artifacts cache"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

https://github.com/cloudfoundry/cloud_controller_ng/issues/2991 https://github.com/cloudfoundry/cloud_controller_ng/issues/2991 When pushing an application you might encounter staging error indicating "Failed to upload payload for build artifacts cache" as show below.

2023-09-07T16:12:33.225-07:00 [API/1] [OUT] Updated app with guid f55f32b7-c980-49f7-bec3-e698bbccb71b ({"buildpack"=>"java_buildpack", "disk_quota"=>1024, "environment_json"=>"[PRIVATE DATA HIDDEN]", "health_check_http_endpoint"=>"", "health_check_type"=>"port", "instances"=>1, "memory"=>1024, "name"=>"sso-management-console", "space_guid"=>"1ceabfc1-b414-46b0-94cb-1b5bf4130404"})
2023-09-07T16:12:35.416-07:00 [API/5] [OUT] Uploading bits for app with guid f55f32b7-c980-49f7-bec3-e698bbccb71b
2023-09-07T16:12:51.525-07:00 [API/4] [OUT] Creating build for app with guid f55f32b7-c980-49f7-bec3-e698bbccb71b
2023-09-07T16:12:51.751-07:00 [API/4] [OUT] Updated app with guid f55f32b7-c980-49f7-bec3-e698bbccb71b ({"state"=>"STARTED"})

2023-09-07T16:12:53.144-07:00 [STG/0] [OUT] Downloading build artifacts cache...
2023-09-07T16:12:53.144-07:00 [STG/0] [OUT] Downloading app package...
2023-09-07T16:12:54.259-07:00 [STG/0] [OUT] Downloaded app package (44.6M)
2023-09-07T16:13:08.652-07:00 [STG/0] [OUT] Downloaded build artifacts cache (956M)
......
2023-09-07T16:14:18.751-07:00 [STG/0] [OUT] Exit status 0
2023-09-07T16:14:18.751-07:00 [STG/0] [OUT] Uploading droplet, build artifacts cache...
2023-09-07T16:14:18.751-07:00 [STG/0] [OUT] Uploading droplet...
2023-09-07T16:14:18.751-07:00 [STG/0] [OUT] Uploading build artifacts cache...
2023-09-07T16:14:21.453-07:00 [API/2] [OUT] Creating droplet for app with guid f55f32b7-c980-49f7-bec3-e698bbccb71b
2023-09-07T16:14:22.425-07:00 [STG/0] [ERR] Failed to upload payload for build artifacts cache
2023-09-07T16:14:30.934-07:00 [STG/0] [OUT] Uploaded droplet (225.1M)
2023-09-07T16:14:30.970-07:00 [STG/0] [ERR] Uploading failed
2023-09-07T16:14:31.600-07:00 [STG/0] [OUT] Cell db1f878c-1b48-4fb3-a49f-d183a10fcec6 stopping instance dbfdd673-acf2-4936-985e-38d262ed99ee
2023-09-07T16:14:31.600-07:00 [STG/0] [OUT] Cell db1f878c-1b48-4fb3-a49f-d183a10fcec6 destroying container for instance dbfdd673-acf2-4936-985e-38d262ed99ee
2023-09-07T16:14:31.695-07:00 [API/0] [ERR] Failed to stage build: staging failed
2023-09-07T16:14:33.139-07:00 [STG/0] [OUT] Cell db1f878c-1b48-4fb3-a49f-d183a10fcec6 successfully destroyed container for instance dbfdd673-acf2-4936-985e-38d262ed99ee

Error "Failed to upload payload for build artifacts cache" indicates the limit set with property Maximum file upload size (TAS 3.0 and earlier versions) or Maximum staged droplet size (TAS 4.0+) is reached.  This property determines the max upload size that TAS will accept for droplets + buildpack cache files.

Environment

Product Version: 2.11

Resolution

There are several ways to address this issue.

• Increase property value of Maximum file upload size (TAS 3.0 and earlier versions) or Maximum staged droplet size (TAS 4.0+). But this is just a workaround and doesn't directly solve the problem. The new limit will still be reached with more droplets and buildpack files being cached
• Delete the application and push it again. All cached data for the app will also be deleted along with deletion of application
•  Clean up the buildpack cache with  cf curl -X DELETE /v2/blobstores/buildpack_cache. This will remove buildpack cache for applications on the platform. Refer to document for details. There is no way to clean up buildpack cache for some particular application at the moment. A feature request has been submitted to community to address the issue.