Aria Operations 8.18.x - Cloud Proxy’s logs/casa-webapps/catalina.out was continually growing to multiple GBs in size, filling the logs/ partition.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Cloud Proxy instances in Aria Operations 8.18.x may experience significant disk space consumption due to an unexpectedly large catalina.out log file 

• Log entries within catalina.out indicate errors as shown below: tail -n catalina.out

ERROR StatusLogger Unable to create Lookup for bundle
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.ResourceBundleLookup

ERROR StatusLogger Unable to create Lookup for ctx
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.ContextMapLookup

ERROR StatusLogger Unable to create Lookup for java
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.JavaLookup

ERROR StatusLogger Unable to create Lookup for log4j
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.Log4jLookup

ERROR StatusLogger Unable to create Lookup for lower
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.LowerLookup

Environment

Aria Operations 8.18.x

Cause

In some rare instances, the casa.war file might not extract correctly into the casa/ directory. While the precise reason for this is still under investigation, removing the existing casa/ directory and restarting the vmware-casa service forces a re-extraction of the casa.war file, typically resolving the issue.

Resolution

Snapshot each nodes, prior making the below changes. 

1. ssh onto the cloud proxy using 'root' user
   
   
2. `cd /usr/lib/vmware-casa/casa-webapp/webapps`
   
   
3. This directory should have two items: a directory named 'casa' and a file named 'casa.war'
   
   
4. `service vmware-casa stop`
   
   
5. Delete the casa/ directory (`rm -rf casa/` or just rename it `mv casa/ casa.bak` if you want to keep a backup)
   
   
6. `service vmware-casa start` (this should re-create the casa/ directory)

Additional Information

Aria Ops has multiple .war files used for various purposes (like the API, the UI, etc.).  Since this catalina.out file is in the logs/casa-webapp/ directory, we know it's the casa.war file having trouble. 

Also, all of the ClassCastExceptions in the catalina.out log have "at org.apache.catalina.startup.HostConfig.deployWAR" in their stacktrace as part of the startup.  This is what's making us think the deployment of the war file was corrupted, leading to either missing or corrupted .jar files in the casa/ webapp's WEB-INF/ directory.

This is still a safe operation and will help us remediate the problem.