UAA VM is out of disk space

Products

• Operations Manager

Issue/Introduction

Symptoms:

The following error is received when running `Apply Changes` in Elastic Runtime:

Started updating job uaa-partition-abc123cf2e36069de2b6 > uaa-partition-abc123cf2e36069de2b6/0 (1a101234-d3ae-4459-9bad-fe7d62bfd1f2) (canary). Failed: Action Failed get_task: Task mad7e5cd6-7b4d-430a-68d0-eb0201d3acdf result: Applying: Applying job uaa: Preparing job: Getting temp dir: mkdir /var/vcap/data/tmp/bosh-agent-applier-jobs-RenderedJobApplier-Apply987994095: no space left on device (00:00:03) 

Environment

Cause

The likely cause of this error is that there is no space left on your UAA partition. You can check this by running the following commands:

• bosh ssh 

To connect to your UAA instance(s).

• df -h

Check that your volumes have sufficient space. You may notice that the /var/vcap/data mount point has used 100% of its disk space and there is no capacity for additional logs on the SDB2 volume.
/dev/sdb2 1003M 987M 0 100% /var/vcap/data.

 

Resolution

Connect to your UAA VM and check if the logs are using all the space available:

• bosh ssh into your UAA VM
• cd /var/vcap/sys/log/uaa

(/var/vcap/sys is a mount point to /var/vcap/data)

List the files in the folder by running the following:

/var/vcap/sys/log/uaa$ ls -al
total 163676

drwxr-x--- 2 vcap vcap     4096 Nov 23 09:17 ./
drwxr-x--- 7 root vcap     4096 Oct 11 12:10 ../
-rw-r--r-- 1 vcap vcap     2309 Oct 11 12:11 catalina.log
-rw-r--r-- 1 vcap vcap        0 Oct 11 12:10 host-manager.log
-rw-r--r-- 1 vcap vcap 45453035 Nov 23 09:44 localhost_access.log
-rw-r--r-- 1 vcap vcap 52448994 Nov  3 14:17 localhost_access.log.1
-rw-r--r-- 1 vcap vcap     2113 Oct 11 12:10 localhost.log
-rw-r--r-- 1 vcap vcap        0 Oct 11 12:10 manager.log
-rw-r----- 1 root root      317 Oct 11 12:11 post-start.stderr.log
-rw-r----- 1 root root      183 Oct 11 12:11 post-start.stdout.log
-rw-r----- 1 vcap vcap     5882 Oct 11 12:10 pre-start.stderr.log
-rw-r----- 1 vcap vcap        0 Oct 11 12:09 pre-start.stdout.log
-rw-r--r-- 1 vcap vcap  1433926 Nov 23 09:44 uaa.log
-rw-r--r-- 1 vcap vcap 53743767 Nov 23 09:17 uaa.log.1
-rw-r--r-- 1 vcap vcap  2343740 Nov 22 16:17 uaa.log.2.gz
-rw-r--r-- 1 vcap vcap  2359000 Nov 21 23:17 uaa.log.3.gz
-rw-r--r-- 1 vcap vcap  2341809 Nov 21 06:17 uaa.log.4.gz
-rw-r--r-- 1 vcap vcap  2342246 Nov 20 13:17 uaa.log.5.gz
-rw-r--r-- 1 vcap vcap  2338044 Nov 19 20:17 uaa.log.6.gz
-rw-r--r-- 1 vcap vcap  2746129 Nov 19 03:17 uaa.log.7.gz

In the event that the logs are filling your /var/vcap/data volume, you have three possible solutions to this issue: A) Short-term, B) Medium-term, C) Longterm:

A) Connect to the UAA VM and check if the logs are using all the space available:

• bosh ssh into your UAA VM
• cd /var/vcap/sys/log/uaa
• rm uaa.log.*

This will delete archived logs from the VM freeing up disk space for other things. 

 B) Scale up your UAA VM to handle the number of logs you are generating:

• Connect to your Ops Manager UI
• Click on your Elastic Runtime Tile
• Click on the resource config tab
• Change the size of your UAA instance. Select a VM with a disk size of minimum 32GB
• Apply Changes

C) Monitor to see if A) or B) fix the problem. If the issue persists, please grab a copy of the UAA logs and open a ticket with PCF support. Support can further assist you with the investigation to see what is causing the elevated UAA logging levels.

Impact

The Impacts/Ricks of carrying out the above actions are low as you are just removing Log Files. If you find that A) or B) are insufficient to fix the problem, then you will want to investigate C) further, as your logs should not keep using all available disk space in this way.