Resolving U00003524 and U00003525 Time Critical DB Calls

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

At its core, the Automic Automation Engine is designed so that every Worker Process (WP) and Communication Process (CP) handles incoming tasks in a serial fashion. Because of this, a server process will wait infinitely for the database to respond to issued statements.

To protect the system and help administrators identify database bottlenecks, server processes automatically log the following errors whenever a database statement takes longer than one second to complete:

• U00003524 UCUDB: ===> Time critical DB call! OPC: '...' time: '1:000.000.000'

• U00003525 UCUDB: ===> Time critical DB call! OPC: '...' time: '1:000.000.000'

For high-performing Automic environments, average database response times should ideally remain between 1 to 3 milliseconds for standard SQL transactions, and 1 to 5 milliseconds for commits. When average response times exceed 50 milliseconds, the system begins experiencing noticeable degradation, often resulting in jobs hanging in a "Generating" status or WP CPU utilization spiking to 100%.

To permanently resolve these time-critical DB call errors, the underlying database responsiveness must be restored to the 1-5 millisecond range. This article outlines Broadcom's best-practice guidelines for Database Administrators (DBAs).

 

Common symptoms of severe database fragmentation include:

• U00003524 or U00003525 (Time Critical DB Call) timeout errors flooding the WP logs.

• Jobs hanging in a "Generating" status for extended periods.

• WP CPU utilization spiking to 90-100% during bulk activations.

• General UI slowness or delayed job processing.

 

Environment

Automic Workload Automation v24/v26

Cause

The most common cause of these U00003524 and U00003525 errors is database table and index fragmentation. The highly volatile nature of Automic workloads - specifically continuous inserts and deletes on E* and M* tables (e.g., EH, EV, ECV, and MQ tables) - causes rapid fragmentation. This leads to stale optimizer statistics and poor query execution plans, forcing database response times past the acceptable thresholds.

Resolution

PREREQUISITES:

• Database Administrator (DBA) access

• Automic Automation Engine administrative access

 

STEPS:

Because Automic Automation Engine performance is heavily dependent on database responsiveness, time-critical database calls must execute efficiently. Please follow the steps below to quantify the errors, perform required maintenance, and rule out network latency.

 

1. Validate and Quantify the Errors (Automic Administrator)

Before engaging your DBA, quantify the errors and identify the specific database operations causing the delay. Run the following command against your AE log directory to extract the error volume and context:

grep -E "U00003524|U00003525" *

(Note: You can pipe this to | wc -l to get a total count).

Review the OPC: value in the output:

• If the OPC is predominantly 'CMIT': The delay is specifically related to storage I/O, network commits, or database log writing. Please refer to Performance degradation or downtime and U00003524 UCUDB: ===> Time critical DB call! OPC: 'CMIT' time: '#:###.###.###'.

• If the OPC shows standard operations (e.g., SLCT, INSR, UPDT): This strongly indicates table/index fragmentation. Provide this grep output to your DBA so they can see exactly which tables and operations are experiencing delays, and proceed to Step 2.

 

 

2. Verify Vendor-Specific Database Configurations

Ensure your database parameters strictly align with Broadcom's official recommendations for your specific database engine. Incorrect configurations (such as improper commit latency settings or isolation levels) directly cause WP exhaustion.

• Recommendations for Oracle

• Recommendations for MS SQL

• Recommendations for DB2

 

3. Perform Routine Database Maintenance

Database table/index fragmentation and stale statistics are the primary causes of jobs hanging in the Generating state. Have your DBA:

• Check for Fragmentation (Bloat): Inspect tables and indexes for fragmentation, paying special attention to highly volatile E* and M* tables (e.g., EH, EV, ECV, and MQ tables). For Microsoft SQL Server, please review Microsoft's procedures on how to check the fragmentation growth of indexes.

• Reorganize/Rebuild Indexes: Rebuild or reorganize fragmented database indexes regularly. The frequency should be dictated by your database vendor's best practices and the volatility of the Automic workload.

  • Best Practice: Many customers report a positive experience scheduling regular index rebuilds on all tables immediately after the Automic REORG and UNLOAD run.

  • Warning: We recommend discussing this matter with your DB-Admin. You might want to schedule the index rebuild at a time of low activity to avoid degraded performance while the rebuild is performed. Please be aware that a certain edition of the database system is required to perform index rebuilds online. If the required license is not available, an index might be taken offline during the rebuild, which will cause severe performance degradation on the corresponding table.

• Update Optimizer Statistics: Ensure database statistics are updated frequently so the database optimizer can efficiently route queries.

  • For Oracle and DB2: Crucially, these statistics must be gathered during periods of typical or high representative load. If gathered when the system is idle, the optimizer assumes volatile MQ tables are permanently empty and will choose incorrect access plans during peak processing, leading to deadlocks. (Automic Admin Action: Provide your typical peak workload windows to your DBA).

  • For MS SQL Server: Rely on AUTO_UPDATE_STATISTICS and standard Microsoft best practices. Manually forcing full statistics updates during peak processing windows on MS SQL can cause severe I/O contention and blocking.

  Verify Schema Integrity Confirm your database objects (tables, indexes, constraints) are in the exact state as defined by the Automic DDL that created them. Missing indexes due to database drift will cause severe performance degradation during bulk job submissions.

 

4. Rule Out Network and Driver Latency

If DBA maintenance is complete but the U00003524 errors persist, the delay may exist in the network layer or JDBC/ODBC drivers rather than the database engine itself. Use the built-in diagnostic tool to measure the transient time between the servers.

• Please refer to: Using ucdbconntest to troubleshoot database slow response time.