BOSH Director persistent disk filesystem is 100% full (500 GB) or becoming full in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The BOSH Director persistent disk filesystem is 100% full (500 GB) or becoming full in Tanzu Application Service for VMs (TAS for VMs). You are unable to run the command "bosh cleanup --all" successfully.

We had to change the size of the persistent disk to 750 GB, and now we are at 60% full. If you review the size of /var/vcap/store/blobstore, you'll see you are currently using 465 GB of space.

Log-bundles stored on the blobstore are retained for 10 days. They are deleted after 10 days any time that new logs are pulled. This retention period is hard-coded and not configurable.

Since there is no easy way to identify and remove log-bundles before the 10-day period, it can be difficult for customers to manage their blobstore space.

This article highlights a script created for customers to clean up log-bundles.

Environment

Product Version: Opsman 2.9, 3.x

Cause

This can happen when there are excessive create_vm failures

Resolution

We created a script for customers to clean up log-bundles and submitted a feature request to make log-bundle retention configurable.

Here is the relevant code, fetch_logs.rb, where the retention period is set.

In order to run the script, you must be logged into the BOSH Director VM as the 'vcap' user.

Script

while read -r id
do
/var/vcap/packages/postgres-10/bin/psql -th 127.0.0.1 -p 5432 bosh -c "delete from log_bundles where blobstore_id = '"$id"';"
  if [ -f /var/vcap/store/blobstore/store/??/"$id" ]
    then
      rm /var/vcap/store/blobstore/store/??/"$id"
  fi
done < <(/var/vcap/packages/postgres-10/bin/psql -th 127.0.0.1 -p 5432 bosh -c "select blobstore_id from log_bundles where log_bundles.timestamp <= now() - interval '5 days';" | sed '/^$/d;s/^ //')

Additional Information

Once the reason for the excessive create_vm failures is fixed, the blobstore should free up some space automatically within the next 10 days