Monitoring tool reporting HEAP Alerts gateway JVM over 90 percent

Products

• CA API Gateway

Issue/Introduction

After upgrading the Gateway appliance from 11.0 to 11.1.1 the gateway JVM started generating heap memory alerts these alerts hit 99 % HEAP 

Environment

Gateway 11.1.1 updated JVM 17

OpenJDK Runtime Environment Temurin-17.0.14+7

Cause

Policy fragment call on all requets implemented “Store to Cache assertions” with a high value for “Maximum entry age”  10 minutes combined with high volume of traffic

Results in objects being moved from Young Generation to Old Generation memory space.

This use case the Gateway JVM set to 21 GB the default configuration cleanup Old Gen when it reaches 70% of the memory 

Old Gen hits over  ~14GB 

Young Gen allocated to reach ~14 

Alerts occur just before Old Generation max is hit then cleanup

Analysis GC logs:  Universal GC Log Analyzer

Resolution

Modify: /opt/SecureSpan/Gateway/runtime/etc/profile.d/appliancedefs.sh

Change from the default 70 to 50

Before

NODE_OPTS="-Xss256k -XX:+UseG1GC -Xmx${java_ram}k -Xms${java_ram}k -XX:G1HeapRegionSize=${G1HeapRegionSize} -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:InitiatingHeapOccupancyPercent=70 -XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=60"

After:

NODE_OPTS="-Xss256k -XX:+UseG1GC -Xmx${java_ram}k -Xms${java_ram}k -XX:G1HeapRegionSize=${G1HeapRegionSize} -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:InitiatingHeapOccupancyPercent=50 -XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=60"

Requires restart of the gatweway process

Old Generation memory will trigger the clean up around 10 GB instead of ~14 GB previously observed 

 

Additional Information

Analyiss the ssg_gc logs in folder /opt/SecureSpan/Gateway/node/default/var/logs using the site 

Analysis GC logs:  Universal GC Log Analyzer

Review the value Old Generation peak, if the percentage is ~70% of heap memory this solution applies