Diego cell VM's ephemeral disk full due to garden not routinely cleaning up its backing store over time

Products

• VMware Tanzu Application Service

Issue/Introduction

Intermittently Diego cell VM is with 99% Disk ephemeral Utilization. But the rep thinks much more space of Disk Available.

The 10GB of storage used by grootfs’s backing store looks strange. 

# du -hcxd1 /var/vcap/data
12K     /var/vcap/data/garden-cni
8.0K    /var/vcap/data/route_emitter
4.0K    /var/vcap/data/vxlan-policy-agent
12K     /var/vcap/data/metrics-agent
1.5M    /var/vcap/data/garden
329M    /var/vcap/data/sys
1.5M    /var/vcap/data/nsx-node-agent
20K     /var/vcap/data/tmp
8.0K    /var/vcap/data/blobs
6.2G    /var/vcap/data/packages
8.0K    /var/vcap/data/syslog_forwarder
20K     /var/vcap/data/root_opt
4.0K    /var/vcap/data/scanner_result
16K     /var/vcap/data/lost+found
10G     /var/vcap/data/grootfs
4.0K    /var/vcap/data/cni
2.8M    /var/vcap/data/bpm
1.2G    /var/vcap/data/dynatrace
4.0K    /var/vcap/data/cfdot
8.0K    /var/vcap/data/metrics-discovery-registrar
8.0K    /var/vcap/data/loggregator_agent
4.0K    /var/vcap/data/cflinuxfs4-rootfs-setup
4.0K    /var/vcap/data/container-metadata
4.0K    /var/vcap/data/dynatrace-oneagent
8.0K    /var/vcap/data/prom_scraper
4.0K    /var/vcap/data/silk-daemon
4.0K    /var/vcap/data/bosh-dns
4.0K    /var/vcap/data/bosh-dns-aliases
140M    /var/vcap/data/root_log
8.0K    /var/vcap/data/loggr-syslog-agent
16K     /var/vcap/data/bosh-dns-adapter
4.0K    /var/vcap/data/nfsv3driver
4.0K    /var/vcap/data/silk-cni
12K     /var/vcap/data/volumes
12K     /var/vcap/data/rpc_statd
4.0K    /var/vcap/data/fim
24K     /var/vcap/data/voldrivers
4.0K    /var/vcap/data/oscap
4.0K    /var/vcap/data/netmon
4.0K    /var/vcap/data/silk-datastore-syncer
8.0K    /var/vcap/data/loggr-udp-forwarder
4.0K    /var/vcap/data/root_var_opt
14M     /var/vcap/data/jobs
4.0K    /var/vcap/data/mapfs
4.0K    /var/vcap/data/config_scanner
12G     /var/vcap/data/rep
576K    /var/vcap/data/containerd
8.0K    /var/vcap/data/loggr-forwarder-agent
4.0K    /var/vcap/data/loggr-system-metrics-agent
4.0K    /var/vcap/data/harbor-dns-aliases
8.0K    /var/vcap/data/sku-name-indicator
8.0K    /var/vcap/data/iptables-logger
4.0K    /var/vcap/data/openvswitch
8.0K    /var/vcap/data/sensitive_blobs
12K     /var/vcap/data/root_tmp
4.0K    /var/vcap/data/smbdriver
4.0K    /var/vcap/data/cflinuxfs3-rootfs-setup
225M    /var/vcap/data/antivirus
30G     /var/vcap/data
30G     total

Cause

There was a bug fixed with garden not routinely cleaning up its backing store over time. This bug was fixed in 4.0.20.

Resolution

At this point, we would advise scaling the disk size up, and also upgrading to a recent version of TAS, as that bug is likely causing garden to not clean up data it thinks isn’t being used anymore, resulting in the discrepency seen in rep’s reported disk space, and actual disk space used by apps.