A bug in PCF 1.7 feeds the executor_cache until the disk is full

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

executor_cache is a place where the cell rep caches build packs, droplets, and the lifecycle binaries. If any of these is missing, it should just download them again. There is a bug that under certain circumstances duplicates the directories, and the zip files are placed in the executor_cache.

The size of the executor_cache can then possibly exceed the specified cache size till disk becomes full. As a consequence, you could experience staging failures when pushing new applications. Besides staging failure, you could also notice this problem by checking the Status page in Pivotal Application Service.

Environment

Resolution

This bug is fixed in 1.7.38, which was released on Nov 24th, 2016. You can download and upgrade to Pivotal Application Service 1.7.38 or higher to fix this issue.

Besides upgrading the Pivotal Application Service, you can also restart the Diego cell to clean up the executor_cache as a workaround. For example:

bosh restart diego_cell-partition-75437f0a091c320d8ca4 0

Impact

When restarting Diego cell, you will experience the app crash temporarily. By the following command, you can check which apps are running and are affected in a Diego cell. For more details, you could refer to How to find what apps are running in a Diego cell. 

Please replace the IP_address_of_diego_cell with the real one on your side:

curl http://ip_address_of_diego_cell:1800/state|python -m json.tool |grep process_guid|cut -d ':' -f2|cut -c 3-38|xargs -I '<guid>' cf curl '/v2/apps/<guid>'|grep '"name"'|cut -d ':' -f2|sort -u