JCS Load Balance Credits 999.00
Article ID: 243305
Updated On:
Products
Issue/Introduction
IMPS credits for that individual JCS server is reported at 999.00 .
Customer is monitoring the IMPS logs (all IMPS servers) for any JCS credits are at 999.00.
Example: Using Telegraf (local OS agent) and Grafana (UI Display)
http://server.test.com/app/kibana#/discover?_g=(refreshInterval:('$$hashKey':'object:2156',display:'10+seconds',pause:!f,section:1,value:10000),time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(beat.hostname,message),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWij_gdKX_5v8d2jpvZj,key:fields.log_type,negate:!f,type:phrase,value:etatrans),query:(match:(fields.log_type:(query:etatrans,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWij_gdKX_5v8d2jpvZj,key:environment,negate:!f,type:phrase,value:prd),query:(match:(environment:(query:prd,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWij_gdKX_5v8d2jpvZj,key:fields.alert_type,negate:!f,type:phrase,value:load_balancing),query:(match:(fields.alert_type:(query:load_balancing,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWij_gdKX_5v8d2jpvZj,key:message,negate:!f,params:!('(999.00)'),type:phrases,value:'(999.00)'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(message:'(999.00)'))))))),index:AWij_gdKX_5v8d2jpvZj,interval:auto,query:(match_all:()),sort:!('@timestamp',desc))
Background: Prior tickets isolated an issue with the JCS->CCS timeout communication.
A new configuration parameter was enabled with the JCS file: server_osgi_ccs.xml (line 98)
<property name="maxWait"><value>10000</value></property>
This parameter addressed the challenge when the CCS would fail to communicate to the JCS and would be restarted via the JCS.
This addressed the stoppage issue.
New Request:
Review support archives and/or notes for additional configurations or tweaks to the JCS configurations under
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf
That would assist with isolation of the issues with the non-responsiveness of JCS (load balance credits of 999.99)
Environment
Release : 14.3, 14.4
Component : IdentityMinder(Identity Manager)
Cause
Caused by JCS restarting.
Resolution
We would see 999.00 messages when JCS stopped/restarted.
The IMPS server maintains a minimum number of idle connections in the pool irrespective of whether they have expired or not and closes the entire connection pool when it realizes that they have become stale (it occurs when the IMPS server tries to serve a connector server request only) and marks the connector server with 999.00. That's why we see 999.00 messages at different times on different IMPS servers.
1. JCS's restarted at ~2am
2. The 999.00 messages can be seen at different times on different IMPS servers because IMPS servers got connector server requests at different times.
JCS service restarted at:
jcs_daily.log.20220602_001p:2022-06-02 02:26:37,283 45276562 [Timer-1] (com.ca.jcs.jcs-management:com.ca.jcs.ui.server.spring.ManagementService:244) DEBUG - JCS Core is now inactive or this bundle is being stopped
jcs_daily.log.20220602_002p:2022-06-02 02:27:47,442 169377970 [Timer-1] (com.ca.jcs.jcs-management:com.ca.jcs.ui.server.spring.ManagementService:244) DEBUG - JCS Core is now inactive or this bundle is being stopped
jcs_daily.log.20220602_009p:2022-06-02 02:28:41,632 169074354 [Timer-1] (com.ca.jcs.jcs-management:com.ca.jcs.ui.server.spring.ManagementService:244) DEBUG - JCS Core is now inactive or this bundle is being stopped
jcs_daily.log.20220602_009p:2022-06-02 02:30:42,993 92346 [Timer-1] (com.ca.jcs.jcs-management:com.ca.jcs.ui.server.spring.ManagementService:244) DEBUG - JCS Core is now inactive or this bundle is being stopped
jcs_daily.log.20220602_010p:2022-06-02 02:31:08,745 169299212 [Timer-1] (com.ca.jcs.jcs-management:com.ca.jcs.ui.server.spring.ManagementService:244) DEBUG -
JCS Core is now inactive or this bundle is being stopped
999.00 started appearing at ("Requested shutdown of CONPOOL”) : for example,
$ grep -ri "Requested shutdown of CONPOOL" *
Binary file etatrans20220602-0000.log_1014 matches
etatrans20220602-0000.log_1015:20220602:022832:TID=2a4b40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf4e8fc98 [ldaps://server2.test.com:20
etatrans20220602-0000.log_1015:20220602:022840:TID=4ffb40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf566c768 [ldaps://server9.test.com:20
etatrans20220602-0000.log_1015:20220602:023110:TID=cfeb40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf4ee6760 [ldaps://server10.test.com:20
etatrans20220602-0000.log_1015:20220602:023145:TID=4f7b40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf566c768 [ldaps://server8.test.com:20
etatrans20220602-0000.log_1015:20220602:023150:TID=224b40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf53cf3a0 [ldaps://server1.test.com:20
etatrans20220602-0000.log_1064:20220602:060059:TID=5feb40:LDAP :----:----:*:
Requested shutdown of CONPOOL 0xf4a88db0 [ldaps://server2.test.target.com:20
etatrans20220602-0000.log_1064:20220602:060059:TID=5feb40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf58d2f70 [ldaps://server9.test.com:20
etatrans20220602-0000.log_1064:20220602:060059:TID=5feb40:LDAP :----:----:*: Requested shutdown of CONPOOL 0xf4a8ed98 [ldaps://server10.test.com:20
etatrans20220602-0000.log_1064:20220602:060059:TID=5feb40:LDAP :----:----:*:
Requested shutdown of CONPOOL 0xf4a86f40 [ldaps://server1.test.com:20
For the above two servers(1015 and 1064), we can see the 999.00 messages at different times. 1015 received the first connector server request at *2 am and 1064 received it at *6 am.
The other IMPS servers exhibited the same behavior.
This is expected behavior and working as designed.
Feedback
