Diego Cell ephemeral disk full in /var/vcap/data/rep/shared/garden/download_cache/

Products

• VMware Tanzu Application Service
• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Diego Cell VMs intermittently reach 90-100% ephemeral disk utilization.

The directory /var/vcap/data/rep/shared/garden/download_cache/ is identified as the primary consumer of space.

Application staging or deployments fail with StagingError or no space left on device.

There is a discrepancy between the disk space reported by the Rep (cfdot cell-states) and the actual physical disk usage on the VM.

Environment

VMware Tanzu Platform - Cloud Foundry  (TPCF)

Cause

A defect in the Diego Rep and Garden components can lead to the download_cache not being accounted for correctly during garbage collection, or failing to trigger cleanup even when disk thresholds are met. This often manifests when many unique buildpacks or Docker image layers are being downloaded frequently.

A permanent fix is currently in progress to improve download_cache utilization accounting.

Resolution

A permanent code fix is targeted for the August Patch Release. In the interim, follow the workaround below to automate cleanup and prevent cell exhaustion.

Workaround: Automation Script

Implement a cronjob script on the affected Diego Cells to monitor the /var/vcap/data threshold and restart the rep service when usage exceeds a specific limit (e.g., 85%). Restarting the rep service forces a synchronization and can trigger underlying cleanup routines.

Note: The following script should be deployed via a BOSH addon or a manual cron configuration if a temporary fix is needed immediately.

This command will make changes to your system. Review it carefully before running.

Script sample:

## chk_clean_disk.sh

#!/bin/bash

# Define the threshold (%)
THRESHOLD=70
# Define the partition to monitor
PARTITION="/var/vcap/data"

# Get current utilization percentage
USAGE=$(df -h "$PARTITION" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    echo "Disk usage on $PARTITION is at ${USAGE}%, which exceeds ${THRESHOLD}%."
    echo "Restarting the rep service..."

    # Command to restart the rep service (monitored by monit)
    sudo /var/vcap/bosh/bin/monit restart rep
else
    echo "Disk usage on $PARTITION is at ${USAGE}%, within safe limits."
fi
 

 

## run script as a cronjob

 bosh -d cf-<DEPLOYMENT>  ssh 'diego_cell' -c "$(cat chk_clean_disk.sh)"
 

## Sample output:

$ bosh -d cf-####### ssh 'diego_cell' -c "$(cat chk__clean_disk.sh)"
Using environment '##.##.##.## as client 'ops_manager'

Using deployment 'cf-#######'

Task 116176. Done
diego_cell/xxxxxxx: stderr | Unauthorized use is strictly prohibited. All access and activity
diego_cell/xxxxxxx: stderr | is subject to logging and monitoring.
diego_cell/xxxxxxx: stderr | Unauthorized use is strictly prohibited. All access and activity
diego_cell/xxxxxxx:: stderr | is subject to logging and monitoring.
diego_cell/xxxxxxx: stdout | Disk usage on /var/vcap/data is at 71%, which exceeds 70%.
diego_cell/xxxxxxx: stdout | Restarting the rep service...
diego_cell/xxxxxxx: stdout | Disk usage on /var/vcap/data is at 71%, which exceeds 70%.
diego_cell/xxxxxxx: stdout | Restarting the rep service...
diego_cell/xxxxxxx: stderr | Connection to ##.##.##.## closed.
diego_cell/xxxxxxx: stderr | Connection to ##.##.##.## closed.