Troubleshooting Locked ConnectALL Automations

Products

• ConnectALL
• ConnectAll On-Prem

Issue/Introduction

In some situations, an automation will go into a locked state and will not come out of the locked state for an excessive amount of time.  This article aims to assist customers with providing customers some of the common causes, as well as steps to gather needed information to provide to support for further troubleshooting.

Cause

This can be caused due to a number of reasons, and it isn't possible to always determine the cause without additional investigation.

Below are links to articles for some known causes with their solutions

Version(s) Affected	Description
3.0.x.x	Syncing fails and automation getting locked when Workflow Transitions are enabled for Status field
3.2.0.1	Locks when processing comments with URLs and large amounts of text

 

Resolution

If the above articles do not pertain to your situation or do not resolve your issue, it will be necessary to gather additional details for the support team to investigate.

There are three key pieces of information that will need to be gathered while the automation is locked

1. Thread dumps
2. ConnectALL logs
3. Database output

Support may request the above steps be done each time your automation is locked.

After this has been gathered, you may unlock the automation, and then there will be an additional set of thread dumps to gather for a baseline of the properly running automation in your environment

 

Thread Dumps

While the automation is locked, we need 3 separate thread dumps against the core service separated by approximately 5 minutes.  There are a number of ways to get thread dumps, however we have a method outlined in the following KB article:

https://knowledge.broadcom.com/external/article?articleId=280280

 

ConnectALL Logs

1. Hover over the lock icon to determine when the automation was locked and note this time, as support will need this.
2. Navigate to your <CONNECTALL_HOME>/Core/logs directory
3. Gather logs for the locked automation and the ConnectALL.log, ensuring that the logs encompass the time you noted above.
   It may be easier for you to ZIP up your logs directory and supply that entire file to the support time

Database Output

Access your database server and open a SQL console scoped to your ConnectALL database

Run the following SQL query, replacing <Automation Name> with the automation that is locked

SELECT application_link_id,application_lock_status, source_app_lock_status,dest_app_lock_status, processing_count, active_poll_id 
FROM application_link 
WHERE application_link_name='<Automation Name>';

Sample Output
+---------------------+------------------+
| application_link_id | processing_count |
+---------------------+------------------+
|                  10 |              270 |
+---------------------+------------------+

Take the application_link_id value and use it in the following query:

MySQL
SELECT poll_details_id, FROM_UNIXTIME(polled_time/1000), FROM_UNIXTIME(completed_time/1000) FROM poll_details
WHERE application_link_id=<active_poll_id>
ORDER BY completed_time DESC;

Microsoft SQL
SELECT poll_details_id, DATEADD(S, polled_time, '1970-01-01'), DATEADD(S, completed_time, '1970-01-01')
FROM poll_details
WHERE application_link_id=<active_poll_id>
ORDER BY completed_time DESC;

PostgreSQL
SELECT poll_details_id, to_char(to_timestamp(polled_time/1000),'YYYY-MM-DD HH24:MI:SS') as polled_time, to_char(to_timestamp(completed_time/1000),'YYYY-MM-DD HH24:MI:SS') as completed_time
FROM poll_details 
WHERE application_link_id=<active_poll_id>
ORDER BY completed_time DESC;

 

Sample Output
+--------------------------------------+---------------------------------+------------------------------------+
| poll_details_id                      | FROM_UNIXTIME(polled_time/1000) | FROM_UNIXTIME(completed_time/1000) |
+--------------------------------------+---------------------------------+------------------------------------+
| aaaaaaaa-0000-0000-aaaa-000000000000 | 2024-04-26 15:16:03.8810        | 2024-04-26 15:16:10.9540           |
| bbbbbbbb-1111-1111-bbbb-111111111111 | 2024-04-25 20:39:21.8160        | 2024-04-25 20:39:26.6520           |
| cccccccc-2222-2222-cccc-222222222222 | 2024-04-25 20:36:07.7180        | 2024-04-25 20:36:11.1290           |
+--------------------------------------+---------------------------------+------------------------------------+

 

Take the top poll_details_id and use it in the following query:

SELECT COUNT(*)
FROM record_transaction_details
WHERE poll_id='<poll_details_id>';

 

Sample Output

+----------+
| COUNT(*) |
+----------+
|        2 |
+----------+

 

Finally run the following query, using the poll_details_id from above:

MySQL
SELECT record_transaction_id, record_id, record_status, FROM_UNIXTIME(created_on/1000)
FROM record_transaction_details
WHERE poll_id='<poll_details_id>'
ORDER BY created_on ASC;

Microsoft SQL
SELECT record_transaction_id, record_id, record_status, DATEADD(S, created_on, '1970-01-01')
FROM record_transaction_details
WHERE poll_id='<poll_details_id>'
ORDER BY created_on ASC;

PostgreSQL
SELECT record_transaction_id, record_id, record_status, to_char(to_timestamp(created_on/1000),'YYYY-MM-DD HH24:MI:SS') as created_on
FROM record_transaction_details 
WHERE poll_id='<poll_details_id>'
ORDER BY created_on ASC;

 

Sample Output

+--------------------------------------+-----------+---------------+--------------------------------+
| record_transaction_id                | record_id | record_status | FROM_UNIXTIME(created_on/1000) |
+--------------------------------------+-----------+---------------+--------------------------------+
| a0a0a0a0-a0a0-a0a0-a0a0-a0a0a0a0a0a0 | <ID>      | PASSED        | 2024-04-26 15:16:04.6010       |
| b1b1b1b1-b1b1-b1b1-b1b1-b1b1b1b1b1b1 | <ID>      | PASSED        | 2024-04-26 15:16:04.7940       |
+--------------------------------------+-----------+---------------+--------------------------------+

 

Take all of the output from the above queries and supply it to support

 

Unlock the automation

At this point, you may unlock the automation by clicking on the lock icon for the automation on your Dashboard page

If the automation does not unlock, it may be necessary to restart your Core service.

 

Thread Dump Baseline

Once your automation is back to processing normally, please gather three additional thread dumps as performed above and provide this to support.  This will be a baseline for a properly working automation so that we can compare with a locked state and only needs to be done once.