newly installed GW 10.x servers crashes regularly with out of heap space error

Products

• CA API Gateway

Issue/Introduction

After migrating GW 9.x servers to GW10 the new servers regularly crashes with out of heap space errors in the ssg logs  while the load, configuration and memory allocation is the same as before .

Environment

Release : 10.0

Component : API GATEWAY

Cause

This could occur when the gateway is configured for large message sizes and/or  audit of these large request / response messages is enabled.

Resolution

The garbage collection settings were changed to G1GC starting with Gateway version 10.0 to optimize performance; however, Java heap space errors may occur if the files or messages involved are large.

The first step could be to increase the memory for example  from 8gb to 16gb or 16gb to 32gb and monitor if this resolved the problem 

If this error occurs keeps occurring , the garbage collection settings can be changed back to ParallelOldGC (i.e., the collector used in Gateway version 9.4 or earlier).

To revert to ParallelOldGC for garbage collection:

Log into the Gateway as a root user.

Open the following file:

/opt/SecureSpan/Gateway/runtime/etc/profile.d/appliancedefs.sh

Replace the following line:

NODE_OPTS="-Xss256k -XX:+UseG1GC -Xmx${java_ram}k -Xms${java_ram}k -XX:G1HeapRegionSize=${G1HeapRegionSize} -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:InitiatingHeapOccupancyPercent=70 -XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=60"

With the following line:

NODE_OPTS="-Xmx${java_ram}k -Xss256k -XX:+UseParallelOldGC"

Restart the Gateway for the new garbage collection settings to take effect.