Error: GC overhead limit exceeded when REST API in use with AdminUI

Products

• SITEMINDER

Issue/Introduction

Running script with REST API calls to the AdminUI like this:

   #!/bin/bash

       echo "Token :";
       echo $myToken;

       for i in `seq -w 1000`; do myToken=`curl -k -u siteminder:password -X POST https://adminui.training.com:8443/ca/api/sso/services/login/v1/token | grep sessionkey | gawk -F: '{print $2}' | grep "\"" | gawk -F\" '{print $2}'`; curl -k -H "Authorization: Bearer $myToken" -H "Content-Type: application/json; charset=UTF-8" -X POST --data @agent$i.json https://adminui.training.com:8443/ca/api/sso/services/policy/v1/SmAgents -v; done; 

The AdminUI reports:

server.log :

   2022-03-01 10:00:52,248 ERROR [io.undertow.request] (default task-1)
   UT005023: Exception handling request to
   /ca/api/sso/services/policy/v1/SmAgents:
   java.lang.OutOfMemoryError: GC overhead limit exceeded

           at com.ca.siteminder.rpc.type.InStream.nextTree(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.getChildren(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.nextTree(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.getChildren(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.nextTree(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.getChildren(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.nextTree(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.getChildren(Unknown Source)
           at com.ca.siteminder.rpc.type.InStream.nextTree(Unknown Source)

The AdminUI also quickly uses more memory:

  20340 root      20   0 4483684   1.5g  34196 S  19.0  9.5   2:51.12 java
  20340 root      20   0 4517608   1.7g  34200 S  22.0 11.2   2:57.56 java
  20340 root      20   0 4536112   1.8g  34200 S  30.7 11.8   3:00.65 java
  20340 root      20   0 4557700   1.9g  34200 S  23.0 12.3   3:03.53 java
  20340 root      20   0 4618352   2.0g  34204 S 138.0 12.7   3:22.80 java
       

and the batch script hangs here:

    }  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   655  100   655    0     0   5131      0 --:--:-- --:--:-- --:--:--  5117
    #* About to connect() to adminui.training.com port 8443 (#0)
    #*   Trying 10.48.33.234...
    #* Connected to adminui.training.com (10.48.33.234) port 8443 (#0)
    #* Initializing NSS with certpath: sql:/etc/pki/nssdb
    #* skipping SSL peer certificate verification
    #* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
    #* Server certificate:
    #*       subject: CN=adminui.training.com
    #*       start date: Feb 23 10:42:39 2022 GMT
    #*       expire date: Feb 21 10:42:39 2032 GMT
    #*       common name: adminui.training.com
    #*       issuer: CN=adminui.training.com
    > POST /ca/api/sso/services/policy/v1/SmAgents HTTP/1.1
    > User-Agent: curl/7.29.0
    > Host: adminui.training.com:8443
    > Accept: */*
    > Authorization: Bearer eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiZGlyIn0..Xqg5ecCn78Olh5xR.mVhK6LFQE3j5cbJHNi3SteY1c9ysYNtkTL56ol5Q3rMn_Zk0IDkU5nFiJoy8uYJjNtBrcBhzAhLwVopgJkAWVfEts59r_Cz0h5fsEMbINfYMDjdGHW2b6tU3a4YGpycrHGLUuVVh9TI20mihKtqQ9nLMsiVqmXXFiifq-U-_61e7Ax_qOniDliIakWKdYEosXKNCvD3RVM72qHz4CMVa0x9XqnRPsnEaBbvLVG1Yix62-cgyNPuTq-zTIP3tg8UKiH1bn4b0nB317V-dq0bYAJj10En1S6O1vS2jPN7WqWNeMws6JOiratP_0s6gyW72Ppp1cE44DjpAqYI_Ocf4LUJQV_J4WGXtNJ1y0xE2yAJU5NopjMtWroMbz7P5cb3rj9xbqeX9xLlb1SFCEgOJsZXttn1lBojW8VBEH9T7HYB45duJooclfA6RXZjAOZpOPnyZmlMXFOhgR6yUL_QqlVczo1EPfRr5dnnc8TeauAgDQ-Wp71-PDr2cXu5_InDIJFIEvG-dTOEEGJaurAkm7beFNHzfKrZBwpxrJiCBLxM.adGuuu-NkHSHirp-70k2Bw
    > Content-Type: application/json; charset=UTF-8
    > Content-Length: 403
    >
    #* upload completely sent off: 403 out of 403 bytes
    < HTTP/1.1 500 Internal Server Error
    < Connection: keep-alive
    < Content-Type: text/html;charset=UTF-8
    < Content-Length: 80
    < Date: Tue, 01 Mar 2022 10:00:53 GMT
    <
    #* Connection #0 to host adminui.training.com left intact
    <html><head><title>Error</title></head><body>Internal Server Error</body></html>  

 

Environment

 

  AdminUI 12.8SP5 on RedHat 7;
  Policy Server 12.8SP5 on RedHat 7;

 

Resolution

 

Put the code to get the token outside the loop code like this:

   #!/bin/bash

   myToken=`curl -k -u siteminder:password -X POST https://adminui.training.com:8443/ca/api/sso/services/login/v1/token | grep sessionkey | gawk -F: '{print $2}' | grep "\"" | gawk -F\" '{print $2}'`

   if [ -z "myToken" ]; then

       echo "null";

   else

       echo "Token :";
       echo $myToken;

       for i in `seq -w 1000`; do curl -k -H "Authorization: Bearer $myToken" -H "Content-Type: application/json; charset=UTF-8" -X POST --data @agent$i.json https://adminui.training.com:8443/ca/api/sso/services/policy/v1/SmAgents -v; done; 

Each instantiated session to handle a REST API call requires intensive
memory. As such, a single session is necessary when executing a
batch. This session should be renewed every 15 minutes as stated in
the documentation (1).

 

Additional Information

 

(1)

    Policy Object REST APIs

      Each call to the Policy Data API and Policy Migration API requires a
      valid JWT. Use the Administrative Token API to obtain that token.

      The Administrative Token API provides a single call that receives
      the credentials (username and password) of a SiteMinder
      Administrator in the Authorization header. If Basic authentication
      for that administrator account is successful, the API returns a JWT
      containing a session ticket.

      Warning 

      The JWT expires after 15 minutes after which it must be refreshed.

    https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/programming/policy-object-rest-apis.html