JMS Connection Settings/Information and how to debug/troubleshoot

Products

• CA API Gateway

Issue/Introduction

Trying to identify the setting in our JMS config where we are opening a persistent connection to the JMS server and how to get more info on these connections'

 

Environment

Release :

Resolution

The Layer7 API Gateway will close an outbound JMS connection under any of the following conditions:

• the connection has been idle for too long (controlled by io.jmsConnectionCacheMaxIdleTime cluster property)   5 minutes
• the connection is too old (controlled by io.jmsConnectionCacheMaxAge cluster property) 10 minutes
• there are too many open connections (controlled by io.jmsConnectionCacheMaxSize cluster property) default 100 connections  - Available connection pool 

Setting to review to see if they have been modified: 

io.jmsConnectionMaxWait - 5s Above properties as well other JMS properties are outlined in the link below :

io.jmsConnectionIdleTime  detail 5 minutes - Max time an idle JMS connection will stay n the connection pool while idle 

io.jmsSessionMaxIdle default:  8  max sessions that can sit idle in the session pool 

io.jmsSessionPoolSize detail:  8 max # of session that can be allocated by session pool 

io.jmsConnectionCacheMaxAge:  Maximum age for a cached JMS connection. Enter "0" (zero) for no time limit. Value is a time unit.

Default: 10m

 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-1/reference/gateway-cluster-properties/input-output-cluster-properties.html

 

This may help:

Add the following value to the log.levels cluster-wide property:
com.l7tech.server.transport.jms2.level = FINE

Use Cases

Generic Connection Pool Flow Scenario:

1. When a Load request starts gateway creates a number defined in Connection Pool Size in Oubtound JMS Destination. ( you should see "New JMS connection created" message in the ssg log) 
2. When the load is finished, unused connections become Idle (you should see a message similar to "jms2.PooledConnection: Active: 0 Idle: 6") 
3. Eviction job for Idle connections runs every time interval defined in cluster-wide properties io.jmsConnectionTimeBetweenEviction. 
4. If an Idle connection reaches a time set in io.jmsConnectionIdleTime it will be removed from the pool. The gateway log will display something like "Closing JMS connection" 
5. CLuster property io.jmsConnectionEvictionBatchSize controls a number of idle connection removed from the pool at once after they exceed io.jmsConnectionIdleTime. Next time the eviction job runs it will remove another set of connection after they exceed io.jmsConnectionIdleTime.
6. We will not remove all idle connections right away. The pool will keep a number of idle connections defined in "Min Connection Idle" property for the Outbound Connection. 
7. The remaining idle connections will stay in the pool until it reaches io.jmsConnectionCacheMaxIdleTime or io.jmsConnectionCacheMaxAge whichever expires first. (io.jmsConnectionCacheMaxAge timeout is calculated from the creation of the pool, io.jmsConnectionCacheMaxIdleTime timeout is reset every time the connection is used. )
8. Once the io.jmsConnectionCacheMaxIdleTime is reached the existing idle connection int he pool are being removed and the pool is closed. That helps to to avoid the resource leak
9. If the load is started again and the pool contains idle connections the idle connection are used first.

Auditing that can be added 

 

The following log records are generated

Operation	Log Level	Example
Create Pooled JMS Object	FINE	FINE    589 com.l7tech.server.transport.jms2.PooledConnection: Creating new      PooledConnection object... FINE    589 com.l7tech.server.transport.jms2.JmsResourceManager: Created new CachedConnection PooledConnection : OUT_AMQP_01,null/myQueueLookup
Pooled connections are being used	FINEST	FINEST  622 com.l7tech.server.transport.jms2.SingleSessionHolder: Borrowed session com.l7tech.server.transport.jms.JmsBag@6ff44e80
Eviction check task	FINE	2017-11-14T11:29:52.331-0800 FINE    95 com.l7tech.server.transport.jms2.JmsResourceManager: Running CacheCleanupTask... 2017-11-14T11:29:52.331-0800 FINE    95 com.l7tech.server.transport.jms2.JmsResourceManager: Check eviction candidate PooledConnection : OUT_AMQP_01,null/myQueueLookup 2017-11-14T11:29:52.331-0800 FINE    95 com.l7tech.server.transport.jms2.PooledConnection: Active: 0 Idle: 3 OUT_AMQP_01,null/myQueueLookup
Pooled connections being closed due to the Idle time	FINE	FINE    95 com.l7tech.server.transport.jms2.SingleSessionHolder: Closing JMS connection (OUT_AMQP_01,null/myQueueLookup), com.l7tech.server.transport.jms.JmsBag@6d7f4b25