BOSH Director Fails to Compile Packages, "No Space Left on Device"

Products

• Operations Manager

Issue/Introduction

Symptoms:

During an "Apply Changes" run on Operations Manager, the BOSH Director fails to compile packages and returns an error.

Error messages

BOSH Director failed to compile packages:

Error 100: No space left on device @ io_write - /var/vcap/store/director/tasks/13211/cpi

Failed compiling packages:

Failed compiling packages > mariadb/40260d929b64e065a5bdaac45ccaa75c2f26474b: No space left on device @ io_write - /var/vcap/store/director/tasks/13211/cpi

Environment

Cause

When you log in to the BOSH Director and see that there is no space available, run the following command:

/:/var/vcap/store/director# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
udev 4075584 4 4075580 1% /dev
tmpfs 817560 680 816880 1% /run
/dev/sda1 3030960 1462172 1397216 52% /
none 4 0 4 0% /sys/fs/cgroup
none 5120 0 5120 0% /run/lock
none 4087784 0 4087784 0% /run/shm
none 102400 0 102400 0% /run/user
/dev/sdb2 301452400 1319956 284796476 1% /var/vcap/data
tmpfs 1024 44 980 5% /var/vcap/data/sys/run
/dev/sdc1 11111111 11111111 11111111 100% /var/vcap/store

You could see your VM has no space as the disk space is up to:

100% /var/vcap/store

The normal size of the persistent disk is configured while installing the VM. You need to change the persistent disk size per your requirement.

Dependency

Confirm that you have adequate disk space for your upgrades. You need at least 20 GB of free disk space to upgrade PCF Operations Manager and Elastic Runtime. If you plan to upgrade other products, the amount of disk space required depends on how many tiles you plan to deploy to your upgraded PCF deployment.

To check current persistent disk usage, select the Operations Manager Director tile from the Installation Dashboard. Select Status and review the value of the PERS. DISK column. If persistent disk usage is higher than 50%, select Settings and Resource Config, and increase your persistent disk space to handle the size of the resources. If you do not know how much disk space to allocate, set the value to at least 100 GB. 

 

Resolution

1. Go to "Operations Manager Director Tile and then to Resource Config". See below:

2. See the memory that is the Persistent Disk Type. You need to increase the disk space based on your requirement.