AIX Agent v24.4 freezes with jobs in Start Initiated and reports in Transferring during production peak

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

The Agent AIX 24.4.2 HF3 where the majority of the Unix Jobs run is not responding and the jobs are taking much longer to execute, resulting in massive backlogs. Every night around EOD, a set of jobs that run on an AIX agent get stuck with jobs in status "Start Initiated" and  a large volume of reports (e.g., 600+) are in a 'Transferring' status simultaneously.

When this occurs, no new jobs are processed and the only way to restore the service consists in stopping the queue where these jobs run and then once the reports are transferred, start the queue again.

Environment

Agent Unix 24.4.2 HF3 on AIX running on the same server as the Automation Engine 24.4.2 HF1

Cause

Analysis was performed via the case DE184796 but it was impossible to find the cause as the needed traces could not be provided as the issue could not be replicated anymore after upgrading to Agent Unix 24.4.5. Most likely the issue is a combination of changes done in DE200941 and DE184186.

Resolution

Workaround:

On the Unix Jobs that run around the time of the issue, change the following:

Section UNIX - Job Report - change "Store to Database" to "Store to File" (and if not needed the reports in case jobs run ok change from "Generate Always" to "Generate On error only" as displayed below

Solution:

Please upgrade the Unix Agent to the latest available version as several bug fixes were solved that could explain why this cannot be reproduced anymore (24.4.5 or superior).

DE184186 (Starting shell might fail on Unix because of erroneous file descriptor handling) should be the particular fix that completely solved the issue.

DE200941 changes should also have applied to solve the issue with reports in transferring: https://broadcomcms-software.wolkenservicedesk.com/external/article?articleNumber=443664

Should this reproduce again on the latest available version of the Agent Unix, please open a new case referencing this article and provide the following information taken when the issue is ongoing:

1. Two or three heap dumps launched every minute of the agent process when the reports are stuck in transferring status, to do so launch the following command replacing by the pid of the agent process (ucxjlx6 for Linux) :
   jmap -dump:live,format=b,file=heap.bin PID_AGENT_UNIX
   
   
2. The agent log and the JCP log.