User deploying an application with 'cf push example-app', and it failed to start the app with the follow errors:
2024-12-02T11:42:02.487-07:00 [STG/0] [OUT] Exit status 0
2024-12-02T11:42:02.487-07:00 [STG/0] [OUT] Uploading droplet, build artifacts cache...
2024-12-02T11:42:02.487-07:00 [STG/0] [OUT] Uploading droplet...
2024-12-02T11:42:02.487-07:00 [STG/0] [OUT] Uploading build artifacts cache...
2024-12-02T11:42:02.548-07:00 [STG/0] [OUT] Uploaded build artifacts cache (129B)
2024-12-02T11:42:04.152-07:00 [API/11] [OUT] Creating droplet for app with guid #######-####-####-####-########
2024-12-02T11:42:30.091-07:00 [API/27] [OUT] Creating droplet for app with guid
#######-####-####-####-########
2024-12-02T11:42:55.136-07:00 [API/26] [OUT] Creating droplet for app with guid
#######-####-####-####-########
2024-12-02T11:43:18.400-07:00 [STG/0] [ERR] Failed to upload payload for droplet
2024-12-02T11:43:18.427-07:00 [STG/0] [ERR] Uploading failed
2024-12-02T11:43:18.701-07:00 [STG/0] [OUT] Cell
#######-####-####-####-########
stopping instance #######-####-####-####-########
2024-12-02T11:43:18.701-07:00 [STG/0] [OUT] Cell
#######-####-####-####-########
destroying container for instance #######-####-####-####-########
2024-12-02T11:43:18.767-07:00 [API/27] [ERR] Failed to stage build: staging failed
Tanzu Application Service
There can be many reasons for the app deployment failures. Note that this error message 'Failed to upload payload for droplet' in the log output message indicating the diego cell failed to upload the app droplet to the blobstore during staging.
The next step is to check the blobstore status using 'bosh vms --vitals' command, and it shows the foundation is using internal mini server blobstore.
Deployment 'minio-internal-blobstore-
#######
'
Instance Process Persistent
Disk Usage
minio-server/
#######-####-####-####-########
running 95% (1i%)
minio-server/
#######-####-####-####-########
running 95% (1i%)
minio-server/
#######-####-####-####-########
running 95% (1i%)
minio-server/
#######-####-####-####-########
running 95% (1i%)
minio-server/
#######-####-####-####-########
running 95% (1i%)
Bosh ssh into a mino-server instance and check the disk usage
minio-server/########-####-######-##########:~# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 807M 7.1G 11% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 2.9G 1.4G 1.4G 51% /
/dev/sdb2 16G 282M 15G 2% /var/vcap/data
tmpfs 16M 52K 16M 1% /var/vcap/data/sys/run
/dev/sdd1 4.0T 1.8T 2.1T 46% /var/vcap/store_migration_target
/dev/sdc1 2.0T 1.9T 582M 100% /var/vcap/store
You can see there is problem with the minio server instances running out of persistent disk space at 95% usage capacity and /var/vcap/store at 100% use. The staging diego cell failed to upload the droplet to the blobstore due to no available disk space.
Here are few possible reasons for using up or running out of blobstore space:
Note that the Cloud Controller does not track resource cache and buildpack cache blob-types in its database so they don't get cleaned up automatically.
Free up the blobstore disk space with manual clean up the buildpack cache using blobstore API with cf cli command as admin user:
cf curl -X DELETE /v2/blobstores/builpack_cache
Allow the command a few minutes to clean up the buildpack cache, and check the mini server instances persistance disk usage with 'bosh vms --vitals' .
Deployment 'minio-internal-blobstore-
#######
'
Instance Process Persistent
Disk Usage
minio-server/
#######-####-####-####-########
running 60% (1i%)
minio-server/
#######-####-####-####-########
running 60% (1i%)
...
It reporting the disk usage drop from 95% to 60%, and running cf push app completed with no issue.
Additionally, you can try free up more blobstore space by manually cleaning up the resource cache as follows:
Warning: Do not manually delete app package, buildpack, or droplet blobs from the blobstore.