Performance issues with Load Balancing on an Oracle database in Applications Manager

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Error Message :
ErrorMsg: AwE-9999 Internal error (6/20/17 3:33 PM)
Details: 537080 TESTQUEUE queue doesn't exist
java.lang.RuntimeException

Databases configured to use an Oracle RAC or a scan address can utilize load balancing between the different nodes. While this many normally increase performance this is not the case if this is done on the Applications Manager Database.

As per documentation it is recommended that this be turned off. The reason why the application need load balancing to be off is that there are known issues with DBMS pipes. Application Manager processes 'sleep' and 'wake up' on DMBS Pipes. Pipes are instance-specific, so if Applications Manager processes are balanced across multiple instances, then wake-ups issued by one process might not activate the target process. For example, the Automation Engine (Master) might sleep on one of the RAC nodes, but the wake-up happens on another node and so the Master continues to sleep. Processes will wake up eventually, but this is not ideal because of performance.

The type of performance issues we have noted include (but are not limited to):

• Delays in Jobs/Process Flows going from QUEUED to RUNNING/INITIATED status.
• Delays in Jobs going from RUNNING status to FINISHED.
• Slowness in Jobs moving from the backlog to history once FINISHED.
• Changes in objects taking a long time to reflect. An example is inactivating a queue may take several minutes to reflect the change.

Beyond performance issues we have also noted other behaviors within the application when load balancing is enabled. These may include:

• The Automation Engine going into a TROUBLE status after a restart.
• Changes in objects not reflecting at all until the Rmi Server is restarted.
• Jobs going to LAUNCH ERROR status.

Cause

Cause type:
Configuration
Root Cause: Having Load Balancing enabled in a RAC/scan address configuration on an Oracle database can cause performance issues.

Environment

OS Version: N/A

Resolution

Load balancing will need to be disabled on both the client side (in tnsnames.ora) and the server side.

One example of how to do this is below:

• Use the ALTER SYSTEM statement to dynamically alter your Oracle Database instance. For example:

SQL> show user
USER is "SYS"
SQL> alter system set remote_listener ='';

System altered.

After you do this, you will need to restart the Applications Manager processes with stopso and startso commands so that connections will go to only one RAC node.

• Back up and edit the init.ora file for each instance in the RAC setup and remove only the REMOTE_LISTENER lines. Do not remove any other lines. This will require a restart of Oracle instances and Applications Manager processes.

BEWARE:  While this is an example of one way to disable load balancing, an Oracle Database Administrator should always be consulted to make sure it is done correctly.
 

Fix Status: No Fix

Fix Version(s):
N/A

Additional Information

Workaround :
N/A