Diego Cell is running out of disk space

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
An application which attempts to run on a faulty Diego Cell will fail because sufficient actual disk space is not available, resulting in `cf push`, `cf start`, and `cf restart` failures.

Use ` df -h`, the Ops Manager > Status page, or ` bosh instance --vitals`  to confirm that the ephemeral disk for the Diego Cell is full.

Environment

Cause

Multiple factors contribute to excessive Diego Cell disk space usage. The following list details commonly observed factors:

• The Diego Cell ephemeral disk is too small.

Diego Cell requires a definite amount of disk overhead. This overhead includes cached files such as buildpacks, docker images, Diego binaries, and log files. By default, a large amount of ephemeral disk can be lost to swap space. In cases where a Diego Cell has less than 64G of ephemeral disk, Pivotal Support notes that the likelihood of large disk space usage increases.
 

• A large number of buildpacks are installed on the system.

Diego primarily stores and caches buildpacks. A large amount of buildpacks require more items to be stored on disk, exacerbating the problem. Pivotal Application Service (PAS) 2.3x through 2.5x have two stacks, `cflinuxfs2` and `cflinuxfs3`, which nearly doubles the number of buildpacks on system.  
 

• A Diego Cell has not been recreated after upgrading to GrootFS from Garden Shed.

If a Diego Cell is not recreated after upgrading, stale information from Garden Shed is left on disk. Diego disk does not account for stale information, causing the disk to be prematurely exhausted.
 

• Diego disk clean up is disabled or needs to be adjusted.

The Diego Cell will run out of actual disk space if Diego disk clean up is disabled. In addition, if` Clean up disk-space once usage fills disk ` is being used, then ` Reserved amount of Disk (MB) ` may need to be adjusted. 

Note: If you are using GrootFS, ` grootfs.reserved_space_for_other_jobs_in_mb ` may need to be adjusted.

To determine which factor is impacting your Diego Cell, see the Resolution section.

Resolution

To reduce the amount of ephemeral disk space or to prevent it from filling up, Pivotal recommends executing the following directions in order:
 

1. Disable swap space on your Diego Cells. BOSH takes the disk for swap out of your ephemeral disk allocation and automatically increases the size of your ephemeral disks without requiring an additional space on your Infrastructure as a Service (IaaS) datastores. This article discusses the amount of swap space saved by BOSH. This article provides detailed instructions for disabling swap space on your Diego Cells.
2. Increase the ephemeral disk size if your Diego Cells have 32G or 64G of Random Access Memory (RAM). A large number of apps can be deployed on the Diego Cell if you have an equally large amount of RAM, putting more demand on the available disk space. Follow the instructions in this article to create a custom VM type if you are unable to find a VM type in Ops Manager that meets your RAM needs. 
3. Set the default disk quota to 512M - refer to this article for more information. Most apps do not require over 1G of disk space. If the default quota is set to a larger value, uneccessary disk capacity is assigned to apps. Using a lower default reduces overall disk usage while also allowing apps to request more disk space if needed.
4. Reduce the number of buildpacks deployed on your foundation. Use `cf delete-buildpack` to remove old or unnecessary buildpacks which reduces the number of buildpacks that must be cached on a Diego Cell.
5. Specify a buildpack or a set of buildpacks when pushing applications. Unspecified buildpacks cause the platform to undergo detection. During the detection process, a Diego Cell downloads and tries all of the buildpacks, taking additional time and putting more demand on the local disk space.
6. Upgrade to PAS version 2.5x or above. The `cflinuxfs2` stack has been removed in PAS 2.5x which decreases the amount of required disk space and removes all buildpacks related to this stack.
7. Verify that Diego disk clean up is disabled. If it is disabled, enable `Clean up disk-space once usage fills disk` and use the default value for `Reserved amount of Disk (MB)`. If it is enabled, verify that the values for `Clean up disk-space once usage fills disk` match the recommendations provided in this article.
8. Verify that you have recently upgraded and switched to GrootFS from Garden Shed. This upgrade generally occurs during the upgrade from Elastic Runtime (ERT) 1.11x to 1.12x, but can delayed if you opted out of using GrootFS.

Note: You can no longer opt out of GrootFS in PAS 2.3x or above.

If the ephemeral disk continues to fill after attempting the workarounds above, follow the instructions in this documentation to examine the Diego Cell and locate the cause of disk space consumption.


If the information gathered from the steps above are not pointing to the cause of excessive disk usage, please open a ticket with Pivotal Support.