"staged_droplet_uploads" folder is 100% full and Cloud Controller fails in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

/var/vcap/data/cloud_controller_ng/tmp/staged_droplet_uploads gradually fills up to 100% and then fails.

To verify the folder size run the following script:

for j in $(bosh -d cf-DEPLOYMENT vms | grep cloud | grep -v worker |awk '{ print $1 }'); do bosh -d cf-DEPLOYMENT ssh $j -c "sudo du -sh /var/vcap/data/cloud_controller_ng/tmp/staged_droplet_uploads"; done | grep droplet

By replacing cf-DEPLOYMENT with the correct ID, the script displays the size of the folder in each Cloud Controller (CC). 

To verify individually, ssh to each Cloud Controller and check the amount of free space. 

df -h

Check the timestamp of the files in the folder. 

/var/vcap/data/cloud_controller_ng/tmp/staged_droplet_uploads# ls -ltr

This issue can be also captured with Healthwatch where the following metrics is emitted:

system_disk_ephemeral_percent{deployment="cf-XXXX", exported_job="cloud_controller"}

Environment

Product Version: 2.10

Resolution

When /var/vcap/data/cloud_controller_ng/tmp/staged_droplet_uploads gradually fills up to 100%, this can lead to issues related to utilization of the Cloud Controller. It is possible you need to scale the Cloud Controllers. For more details, refer to Scaling Cloud Controller.

Note: When scaling up the Cloud Controllers, make sure to have the existing Cloud Controllers also recreated.

These symptoms were caused because of an error with what the ephemeral storage Cluster API (CAPI) was using or referring to. However, failures to delete a file form in this folder are not logged.

Recreating the VMs fixed this because doing this creates a new ephemeral storage.