Diego cell disks fill up after GrootFS upgrade

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
Pivotal Cloud Foundry (PCF) Foundation was upgraded from 1.11 to 1.12+ version of Pivotal Application Service (PAS) aka Elastic Runtime. This switches the garden filesystem from garden shed to GrootFS.

The ephemeral disk of cell is 100% despite fact that GrootFS is not occupying entire disk space.

Files are observed in Shed’s graph directory (/var/vcap/data/garden/graph/aufs) even with GrootFS enabled. 

Environment

Cause

The difference between these versions is that Garden’s default storage driver changes from Shed to GrootFS, and these each have their own storage location.

The new container rootfs provider (GrootFS) only has knowledge of its own graph, and would not know how to clean up the old provider’s (Shed) graph. When switching to a new storage driver, we ask that operators recreate their cells to avoid leaving stale files cluttering up the data disk.

This is referenced in 1.12 release notes: https://docs.pivotal.io/pivotalcf/1-12/pcf-release-notes/runtime-rn.html

Note: It is recommended that you re-create all VMs when upgrading to this release, due to the update to garden-runc-release. This will happen automatically if you are updating your stemcell. If not, you can check the “Recreate All VMs” checkbox at Director Config on the Operations (Ops) Manager Director.

 

Resolution

It will be necessary to clean up the stale Garden shed files. This can be done either by recreating VM's or via manually cleanup. The recommended solution is to recreate the VM's as this guarantees a clean state for GrootFS.

Recreate all VMs:
This can be done by going to Directory Config at Ops Manager and selecting Recreate all VMs check-box.

Alternatively, you can upgrade the Stemcell version of PAS aka Elastic Runtime (ERT) tile by downloading newer Stemcell from network.pivotal.io. This will also force a recreate of VM's.

Click Apply Changes to begin VM recreation process.

Manual steps:

1.  Confirm that GrootFS is enabled by going to PAS Tile, Settings tab, select Application Containers, and verify that  "Enable the GrootFS container image plugin for Garden RunC" is checked off. 
   
   WARNING: If GrootFS is disabled then DO NOT proceed with these steps.
2. Check the contents of /var/vcap/data/garden/depot/*/config.json on Diego cell and see that each root.path is set to /var/vcap/data/grootfs/store/{unprivileged,privileged}/images/*/rootfs. If all match that pattern, then it is safe to delete the contents of /var/vcap/data/garden/graph/aufs.
    
3. Manually remove the contents of graph aufs directory:
   rm -rf /var/vcap/data/garden/graph/aufs

Afterwards, check capacity of Diego cells with df -h in order to validate space has been reclaimed.