VMware vRealize Automation appliance 7.x reports /storage/log 100% full

Products

• VMware Aria Suite

Issue/Introduction

Explains the cause, troubleshooting steps, notes and explanations, possible resolution, workarounds and recommendations on planning to avoid similar issues from happening in the future.

Symptoms:
VMware vRealize Automation (vRA) appliance /storage/log file system shows as 100% used.

Environment

VMware vRealize Automation 7.x

Cause

Two main scenarios that can cause filling of the/storage/log/vmware files system in are:

1. Improper configuration of log file rotation: If not configured properly, the log rotation can cause large sized files and also numerous log files, which overtime, can consume all the available space. 
   1. Verify the disk space usage for vcac, vco and horizon directories with the following commands from a SSH session/console:
      • du -h /storage/log/vmware/vco
      • du -h /storage/log/vmware/vcac
      • du -h /storage/log/vmware/horizon
2. Creation of large Java Memory dump (.hprof ) files:
   • vRealize Orchestrator can run into an OutOfMemory exception based upon configuration and system usage which will dump an hprof file to disk.  This can subsequently cause a full drive.

Resolution

1. Fix the configuration of log rotation, see examples here:

Configuring log rotation by file size in vRA and vRO
Note! Open a Global Support Services Service Request if there are any additional questions or concerns regarding this action.

2. Allocate more space for the vRealize Automation log file system and/or for Java Heap dump (.hprof) files.
   Note: If a vco.hprof file is created repeatedly within /storage/log/vmware/vco/app-server this most likely indicates that vRO is under heavy load.  The first step should be to increase the Java heap size or add more space to the associated volume/disk.Workaround:
3. Delete the older log files to ensure there is available space for new log activity.
4. Delete  /storage/log/vmware/vco/app-server/vco.hprof heap dump files and write a script file to delete the vco.hprof files periodically.

Note! hprof files are heap dumps in binary format, which can be used for detailed analysis of memory related problems in the Java stack.  These are safe to delete and are not actively in use by a healthy running system.
Note! A cron job can be created to run on a schedule to check for large *hprof files on the system to be removed.

Additional Information

Notes and explanation:
 1. Log files size and rotation configuration:
Depending on the level of server log and the scripting log the system is configured with, and how often these logs are generated, the file can grow in size substantially. 
The default log level of the server log and the scripting log is INFO. Changing the log level affects all new messages that the server enters in the logs and the number of active connections to the database. The logging verbosity decreases in descending order.

Recommendations: Only set the log level to DEBUG or ALL to debug a problem. Do not use these settings in a production environment because it can seriously impair performance.
To prevent the server log from becoming too large, you can set the maximum file size and count of the server logs by modifying the values in the Max file count and Max file size (MB) text boxes. 
 
The above document suggests to set the maximum size of the log file (Ex 5M). Once it exceeds the set size it will rotate 20 times and after that it compresses the old file and creates a new one.

Impact/Risks:

• When the log file system is full, newer log messages will not be properly recorded within the associated log files, (catalina.out, vcac-config.log, controlcenter.log, localhost_access_log.txt, horizon.log... etc)  which are required to be written to by the underlying applications and services.
• The log directories mentioned above are located under the Linux root directory--filling up this volume can cause serious issues with system stability and services.