Ops Manager Director ephemeral disk full "No Space left on the device"

Products

Operations Manager

Issue/Introduction

Symptoms:

The Operations (Ops) Manager Director VM ephemeral disk (/var/vcap/data) becomes full and an error message similar to the one below can be seen in Ops Manager log files.

p[2017-05-26 06:52:22 #20887] [task:5425] ERROR -- DirectorJobRunner: 
Failed to extract release archive '/var/vcap/data/director/tmp/0000000221' into dir '/var/vcap/data/tmp/director/d20170526-20887-ge9jfr', 
tar returned 2, output: tar: ./compiled_packages/golang1.7.tgz: Wrote only 8192 of 10240 bytes 
tar: ./compiled_packages/cc_uploader.tgz: Cannot write: No space left on device

Confirm the disk is full by ssh vcap@<director vm>. Use the VM credentials from the Ops Manager Director GUI credentials tab.

Then run df -h /var/vcap/data and confirm its full.

For example:

/:~# df -h /var/vcap/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb2        46G  46G   0G   100% /var/vcap/data

Environment

Cause

The cause of this issue is that the ephemeral disk, mounted at /var/vcap/data/ on the Ops Manager Director VM, has become full.

This is known based on the error message "no space left on device" which is see when something attempts to write to a path beneath /var/vcap/data.

While the reason for the disk being full can vary, it is commonly caused when there is higher than expected logging volume which fills up the disk. If you'd like to find the culprit, you can run the following command:

$ du -h -d 1 /var/vcap/data to see which subdirectory is consuming the disk space.

Resolution

There are two general ways to resolve this issue:

1. Delete files to free up disk space

2. Increase the disk space.

See the sections below for each solution.

Delete files

In general, we do not recommend deleting files from the Ops Manager Director VM, however, if you are running out of disk space because of log files, you can safely delete log files to free up space. Here are the steps to do that.

1. SSH to the Ops Manager Director VM. You can get the IP from Ops Manager > Director > Status and you can get the VM credentials from Ops Manager > Director > Credentials, both in the Ops Manager Web UI.

2. On the Director VM, run the following command to get a root shell:

$ sudo su

Then run the following:

# cd /var/vcap/sys/log

3. Before you delete log files, it is recommended that you scp the log files off of the Director VM to another system. This will allow for further analysis of the logs to see why they are filling up the disk.

In general, logs should not fill up the ephemeral disk so it's important to understand why it is happening to prevent this from happening again in the future. If you need assistance with this task, please open a ticket with Pivotal Support.

4. To continue deleting the logs, run the following:

# monit stop all

This will instruct all of the processes on the Director VM to begin shutting down. Also run the following and wait for all of the processes to stop:

# watch monit summary

5. Run the below command to delete all of the log files:

# rm -rf /var/vcap/sys/log/*

6. Run the following two commands in order and wait for all of the processes to start.:

# monit start all

# watch monit summary

At this point, all of the Director processes should be running again and you should have free disk space.

You can verify this by running:

# df -h /var/vcap/data

Increasing Disk Size

To increase the ephemeral disk for the Ops Manager Director, go to the Ops Manager Web UI.

Navigate to Ops Manager Director > Settings > Resource Config and change the VM type in Ops Manager Director row so that it has a larger ephemeral disk.

Click the "Save" button. Navigate back to the Ops Manager Dashboard. Then click "Apply Changes".

This will redeploy the Ops Manager Director VM with a larger disk.

Note: In general, we recommend that you pick the "Automatic" option for the VM Type, which is the default VM type and allows Ops Manager to update the VM type to the current recommendation from Pivotal as you upgrade to newer versions of Ops Manager.