BOSH Director persistent disk fills up due to accumulated blobstore content

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Platform

Issue/Introduction

The BOSH Director persistent disk (/var/vcap/store) reaches high utilization or 100% usage and disk usage analysis shows that the BOSH Director internal blobstore directory (/var/vcap/store/blobstore) consumes the majority of the available space on the Director persistent disk when inspected using:

bosh/0:~# du -sh  /var/vcap/store/*

 

This indicates that accumulated blobstore artifacts are the primary contributor to persistent disk exhaustion.

Environment

BOSH Director on Tanzu Platform

Cause

The BOSH Director stores the following artifacts in its internal blobstore:

• Uploaded release versions

• Uploaded stemcells

• Compiled packages (per release + stemcell combination)

• DNS record blobs

• Exported releases

Over time, especially in environments with:

• frequent upgrades

• stemcell churn

• tile redeployments

• certificate rotations

• Windows and Linux stemcells

 

the Director accumulates unused releases, stemcells, and compiled packages.

These artifacts are not automatically garbage-collected unless an explicit cleanup is performed.

Resolution

Cleanup command options

The BOSH Director provides two cleanup modes with different levels of aggressiveness.

bosh clean-up

Runs a conservative cleanup removes unused resources but keeps orphaned disks and the two most recent versions of stemcells and releases.

This option is recommended as a first step when reclaiming disk space on the Director.

bosh clean-up --all

Runs a full cleanup that removes all unused resources (including orphaned disks)  such us releases, stemcells, and compiled packages that are not referenced by any active deployment .

This option reclaims the maximum amount of disk space and is recommended when the default cleanup does not free sufficient capacity.

You can combine it with --keep-orphaned-disks in order not to delete orphaned disks. 

Recommendation

It is recommended to:

1. Run bosh clean-up initially and verify recovered disk space.

2. If disk usage remains high, run bosh clean-up --all --keep-orphaned-disks to perform a full cleanup while preserving orphaned persistent disks.

 

Detailed Procedure

Step 1: Validate cleanup impact (recommended)

Run the cleanup command in dry-run mode:

 

bosh -e <environment> clean-up --dry-run (or --all --keep-orphaned-disks --dry-run for potencial full garbage collection)

This command lists:

• unused releases

• unused stemcells

• unused compiled packages

• stale DNS blobs

No data is deleted in dry-run mode.

Step 2: Perform cleanup

If the dry-run output is acceptable, run:

 

bosh -e <environment> clean-up (or --all --keep-orphaned-disks for full garbage collection)

This operation:

• deletes only artifacts not referenced by any active deployment 

Additional Information

Important Notes

• bosh clean-up never deletes artifacts referenced by active deployments

• Compiled packages are fully reproducible and safe to remove

 

Verification

After cleanup, verify reclaimed space:

df -h /var/vcap/store 
du -sh  /var/vcap/store/*

 

 

Prevention / Best Practices

• Run bosh clean-up periodically in long-lived foundations