Multiple Masters enter an "elapsed state'

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

In Applications Manager environments, multiple masters may simultaneously enter an "elapsed" state where no jobs or chains execute. This behavior occurs when the RmiServer connection pool encounters a communication timeout with the database, often during subvar resolution or global name lookups.

The Java Client shows masters in an "elapsed state". While this is not an official term, clients have described an elapsed state as when the below value exceeds 1 or 2 minutes with the Master appearing to be in a hung status (nothing processing or updating):

The RmiServer<timestamp>.logs show the following errors:

rfp0: .DBAccess: checkError 17002rfp0: .AxConnectionPool: shutdown (sync) jdbc:oracle:thin:@//####-scan.####.com:1521/####:appsconnCkr: AwE-9999 Internal errorDetails: connection found to be is use for more than 5 minutes and validate method failed, removing connection from pooljava.lang.RuntimeException: jdbc:oracle:thin:@//####-scan.####.com:1521/####:apps

Commonly, the hang occurs on the following query OR can be ANY query: select lower(substr(GLOBAL_NAME,1,7)) from global_name

What is observed is that Applications Manager communicates to it's respective database to run a sql SELECT and it never returns. The same behavior is observed on multiple masters masters located on server in the same and/or different time zones indicating an environment issue.

Environment

CA Automic Applications Manager 9.4 or higher
Oracle 19c

Cause

This is an environmental or network issue. The Applications Manager RmiServer initiates a database query that fails to return a response within the hard-coded 5-minute validation window, causing the connection pool to shut down and the master to stall.

Resolution

1. Restart Services: Restart the Applications Manager processes or services on the impacted master to clear the hung connection pool and restore job processing.
2. Verify Network Path: Work with your network team to inspect the path between the master server and the database scan listener for latency or dropped packets.
3. Check Shared Infrastructure: If multiple masters are impacted, investigate shared components such as a common firewall, load balancer, DNS server, network, or datacenter.

If the issue persists and no network/DB root cause is found, contact support.

To speak with a customer representative or a Support Engineer see Contact Support. 

Additional Information

For more information on how to subscribe to articles for updates, see Article Subscription.