Uncompressed gc.logs are causing the /root partiton to run out of space in VMware Identity Manager.

Products

• VCF Automation

Issue/Introduction

The directory /opt/vmware/opensearch/logs will show gc.log getting rolled-over but not getting compressed, all the gc.log files will be 64m in size except for the recent few.

root@<vidm-fqdn> [ /opt/vmware/opensearch/logs ]# df -h /
Filesystem       Size   Used   Avail   Use%  Mounted on
/dev/sda4       8.8G   7.8G   499M   95%      /

root@<vidm-fqdn> [ /opt/vmware/opensearch/logs ]# ls -ltr
-rw-r--r-- 1 opensearch users 67108884 Nov 05:11 gc.log.05
-rw-r--r-- 1 opensearch users 67108898 Nov 22:21 gc.log.06
-rw-r--r-- 1 opensearch users 67108901 Nov 15:39 gc.log.07
-rw-r--r-- 1 opensearch users 67109004 Nov 08:54 gc.log.08
-rw-r--r-- 1 opensearch users 67108899 Nov 13:35 gc.log.09
-rw-r--r-- 1 opensearch users 67108942 Nov 01:26 gc.log.10
-rw-r--r-- 1 opensearch users 67109025 Nov 18:39 gc.log.11
-rw-r--r-- 1 opensearch users 13313653 Nov 02:44 gc.log.12
-rw-r--r-- 1 opensearch users     1373 Nov 02:44 gc.log.13
-rw-r--r-- 1 opensearch users  1940305 Nov 03:06 gc.log

Environment

VMware Identity Manager 3.3.7

Resolution

By design, this is expected behavior. However, the filesystem from filling up can be prevented by reducing the GC log file size and limiting the number of retained log files. Please refer to the action plan below to address this issue.

Action Plan:

1. SSH into the node as root.
2. Make of copy of the jvm.options file by running the command:

   cp /opt/vmware/opensearch/config/jvm.options /root/jvm.options.bak

3. Edit the jvm.options file by running the following command:

   vi /opt/vmware/opensearch/config/jvm.options

4. Update the existing configuration from:

   # JDK 9+ GC logging 9-:-
   Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

   TO
   

   # JDK 9+ GC logging 9-:-
   Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=11,filesize=5m

5. Save and exit the file using:

   :wq!

6. Restart the Horizon-workspace service followed by Opensearch service.

   service horizon-workspace restart
   /etc/init.d/opensearch restart

   Note: Although the OpenSearch service typically restarts automatically when the horizon-workspace service is restarted, the OpenSearch service must be explicitly restarted for the configuration changes to take effect.

7. Perform the step 1-6 on all the nodes.

Applying the above workaround will reduce both the number and the size of the gc.log files. The existing 64 MB log file will be cleaned up automatically during the next few housekeeping cycles.