Error: PWP not assigning after node failure in Oracle RAC environment - Automic Automation Engine

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

In an Automic Automation environment running on Oracle RAC, an engine server crash may result in the system being unable to initialize a Primary Work Process (PWP). Services may appear 'green' in the Service Manager, but no connections initialize because the database retains sessions from the failed node.

Environment

• Product: Automic Automation Engine
• Version: 24.4 and higher
• Database: Oracle RAC
• Operating System: Windows Server / Linux

Cause

The Oracle database instance retains locks and sessions from the crashed engine server. These 'dead' sessions prevent new service processes from connecting and initializing as the PWP.

Resolution

To ensure the database quickly releases sessions after a node failure, configure Dead Connection Detection (DCD) and specific TCP settings.

1. Enable Dead Connection Detection on the Database Server:

   • Open the sqlnet.ora file.
   • Set SQLNET.EXPIRE_TIME=1 (or a maximum of 1 minute). This prompts the database server to check if client sessions are still alive and automatically closes non-responding sessions.

2. Configure TCP Keepalive at the OS level (Database Client):

   • Set the operating system parameter tcp_keepalive to a value <= 5 minutes. If the database server cannot end a session cleanly, this ensures the database client detects the connection failure and terminates it.

3. Update the tnsnames.ora file on the Database Client:

   • Add (ENABLE=BROKEN) as the first parameter in the DESCRIPTION block.

4. Configure Automation Engine System Settings:

   • In the UC_SYSTEM_SETTINGS variable, set LAST_USE=0 to improve stability in RAC environments.

5. Verify Oracle RAC Initialization Parameters:

   • In the INIT.ORA file, ensure _lm_dd_interval is set to <= 10 seconds (or 0 to identify deadlocks immediately).

⚠️ IMPORTANT: Always test configuration changes in a non-production environment. Take a backup of configuration files before implementing changes in production.

 

See more information/recommendations for Oracle/RAC/DataGuard here: Recommendations for Oracle