Performance degradation or downtime and U00003524 UCUDB: ===> Time critical DB call! OPC: 'CMIT' time: '#:###.###.###'

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation

Issue/Introduction

Sometimes there may be an outage where users are unable to access the AWI or no work being done by the WPs.  The system may recover on its own after some time or may require a complete restart.  

In the WP logs files there are many lines like the following:

U00003524 UCUDB: ===> Time critical DB call!       OPC: 'CMIT' time: '#:###.###.###'

sometimes this is accompanied by:
U00003525 UCUDB: ===> 'commit'

Environment

Release : 12.x, 21.0.x

Cause

This is caused by problems with network, server I/O, or database

Resolution

There is nothing in Automic that causes long commit times as all that is being done is a commit statement is sent to the database from Automic. The problem will be something between Automic and the database, the database itself, or between the database and Automic for the return call.

If this continues to happen, a DBA, system admins, and/or network admins need to troubleshoot what is happening in one of the following places (includes, but is not limited to):

• The database client when the call is sent out from Automic - this can be the database client, jdbc driver, odbc, etc...
• I/O on the Automic server for the call going out to the database
• The network between Automic and the database server (this can include switches, routers, etc...) as the call goes out from Automic to the database
• I/O on the database server as the call comes in from Automic
• Database performance on the database itself
• I/O on the database server as the return leaves the database to go back to the Automic server
• The network between the database server and Automic server as the return goes back to the Automic server
• I/O on the Automic server for the call returning to Automic
• The database client on the Automic server as the return comes back to Automic

If the database calls take over 10 minutes on a non-PWP process (WP, CP, JWP, JCP), there could be a disconnection of that process as it is busy waiting for a response on the commit statement.
If the database call takes over 10 minutes on the PWP process, the entire system could be inaccessible as the process is still running and is unable to respond to any other process.

The following article:  How to improve overall Automation Engine performance mentions some information around long commits but it is more pointing out that commits should not take longer than 5 ms than anything else.

Hopefully this helps show some information about what was happening on the system when the issue occurs.  The root cause for the application issues are the long commit calls, beyond that, an admin needs to be involved to say why I/O was having problems during the time