Jobs take too much time to complete after script end

Products

• CA Automic Dollar Universe

Issue/Introduction

Since upgrading a reference node AIX from version 5.6 to version 6.10.61, the customer has noticed a drift in the required time to end a job during their nightly batch, starting around 21:00. For example, a job whose script has ended at 21:04 will have the completion instruction complete at 21:16.

Since this is AIX, many tuning instructions were applied  but the problem reocurred at night ( during production peak).

These are the variables added for tuning reasons:

<var id="U_RES_AVAIL_AREAS">X</var>
 <var id="U_DQM_THREAD_STACK_SIZE">256</var>
 <var id="UNI_AIX_LDR_CNTRL_MAXDATA_IO">MAXDATA=0x60000000</var>
 <var id="U_TIMEOUT_REORG">2</var>
 <var id="U_IO_THREADPOOL_MAX">1024</var>
 <var id="UNI_CACHE_INIT_XNAMES">4000</var>
 <var id="UNI_CACHE_INIT_OBJECTS">650</var>
 <var id="U_BVS_THREAD_STACK_SIZE">256</var>
 <var id="U_CDJ_THREAD_STACK_SIZE">256</var>
 <var id="UNI_AIX_LDR_CNTRL_MAXDATA_CDJ">MAXDATA=0x20000000</var>
 <var id="UNI_AIX_LDR_CNTRL_MAXDATA_BVS">MAXDATA=0x20000000</var>

Customer has noticed that UVC is speedier, but the time drift remains. This impacts the majority of jobs after 21:00, though some wait longer times between script completion and completion instructions than others. 

We don't see any uxjobend processes that are dragging on, what could be the cause?

Environment

Release : 6.x

Component :Dollar Universe

Cause

Faulty U_POST_UPROC, this could be seen by adding some date commands in the U_POST_UPROC and checking the Job Log, to check how long it would take each command to complete.

Resolution

The issue due to the U_POST_UPROC that was far from optimal.

On this case, customer was launching 6 Dollar Universe commands (5 uxlst ctl + 1 uxlst upr) for each Uproc execution to fill a file with details about the ongoing execution, this generated an additional overhead to the IO server for nothing during production peak (around 21:00) for nothing.

Instead, a echo in the u_ante_uproc and u_post_uproc regarding the current uproc and timestamps for start and end could be used for monitoring in a txt file if necessary, else use Reporter which does this monitoring natively.

Else,  simplify the U_POST_UPROC script to reduce the 5 uxlst ctl commands to only one or rename the U_POST_UPROC.