Pivotal Application Service upgrade fails due to UAA issue

Products

• Operations Manager

Issue/Introduction

Symptoms:

During the upgrade of Pivotal Application Service, the UAA fails to upgrade due to a transient issue. Looking at the Catalina.log you will see an error similar to the following:

SEVERE: Exception unloading sessions to persistent storage
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1823)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:719)
at org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:445)

Another error you may find in the logs is:

log4j:ERROR Failed to flush writer,
[2016-06-30 11:32:29+0000] java.io.IOException: No space left on device

BOSH agent logs will show something similar to the following:

2016-05-30_10:22:30.53877 ./jdk/lib/amd64/jli/
2016-05-30_10:22:30.53877 ./jdk/lib/amd64/jli/libjli.so
2016-05-30_10:22:30.53878 ./jdk/lib/amd64/libjawt.so
2016-05-30_10:22:30.53878 ./jdk/ASSEMBLY_EXCEPTION
2016-05-30_10:22:30.53878 [Cmd Runner] 2016/05/30 10:22:30 DEBUG - Stderr: tar: ./jdk/jre/lib/amd64/libawt.so: Wrote only 9216 of 10240 bytes
2016-05-30_10:22:30.53878 tar: ./jdk/jre/lib/amd64/libsunec.so: Cannot write: No space left on device
2016-05-30_10:22:30.53879 tar: ./jdk/jre/lib/amd64/libsctp.so: Cannot write: No space left on device
2016-05-30_10:22:30.53879 tar: ./jdk/jre/lib/amd64/libsaproc.so: Cannot write: No space left on device 

Environment

Cause

The reason for this issue is not enough disk space on the virtual machine (VM) for BOSH to install the required packages.  

 

Resolution

• A retry of the upgrade should allow the install to complete successfully.
• Another option is to increase the disk space allocated to the failed VM.