Intermittent Performance Slowness issues with AutoSys

Products

• Autosys Workload Automation

Issue/Introduction

AutoSys Workload Automation performance issues were seen following a migration to cloud based infrastructure, specifically affecting Quick View, the Monitoring tab, and job action execution (e.g., force start or hold). This issue is often characterized by slow response times in the Web UI and delays in processing command prompts.

Symptoms:

• Slow loading of the Monitoring tab in WCC.
• Quick View latency when searching or refreshing.
• Delayed execution of job actions in the UI (to be able to do Force Start, Hold, Terminate).
• Performance warnings in as_server logs:
  CAUAJM_W_10998 Performance warning: Below query execution time has exceeded the DBLibThresholdTime.

Cause

The performance degradation is caused by EEM LDAP configuration set to resolve nested groups.

In cloud or high-latency network environments, traversing nested group hierarchies in an external LDAP user store increases authorization time exponentially. Additionally, pointing to a slow or load-balanced LDAP server network can further exacerbate the delay.

Evidence for such can be determined by enabling debug in both application server and EEM:

• Enable EEM Server side debug first: How to enable debug in EEM (AutoSys)?
  
  
• Enable additional debug on Application Server
  
  
  • Below example is for Linux/Unix,  for Windows, it will be %AUTOUSER%  instead of $AUTOUSER,  %AUTOSERV% instead of $AUTOSERV
  • $AUTOUSER/config.$AUTOSERV  --> make a backup of this file and then edit original file
  • Look for   ISDBGACTIV=OFF
    change it to look like:
    ISDBGACTIV=LIGHT,JOB,DBQUERY,SEC
Note: You cannot should only have one uncommented like ISDBGACTIV  with the above value
  • Save the file
  • ps -few|grep as_server   ## to get PID of as_server
  • kill -HUP <pid from above> ## to enable debug,  
    Note: For Windows, Pause and Resume the Workload Automation Application Server Service 
  • debug information goes to $AUTOUSER/out/as_server.$AUTOSERV  files
    
    
• Symptoms in the logs would be like below,, where the number is in micro seconds:
   
  SEC: EEM::Authorize Matrix: User=#### Total=2
SEC: EEM::Authorize Matrix ret[1] elapsed [15867878]
  
  
• Corresponding EEM performance.log contains entries like:
  
  usersessionbuilder getdirectoryuserdetails [identity: ####, retval: ####] [timetaken: 15854 ms]
userstoremanager resolvenestedgroups [principalname: ####, level: -2, resultset: 4, retval: 1] [timetaken: 292 ms]
  
  

Resolution

• Ensure you are on AutoSys 12.1.x or 24.x latest cumulative patches.
   
• Check with LDAP administrators to see if the external LDAP store is indeed taking much longer to resolve these groups. The LDAP configuration can be found via:
  • Log into the Embedded Entitlements Manager (EEM) UI as eiamadmin
  • Navigate to the Configure tab.
  • Select User Store in the top menu, then click User Store in the left-hand pane.
  • Under the Directory Information tab, click on the directory name to show the details of what LDAP servers it is using. 
  • Work with LDAP admins to narrow down the scope of why the resolutions are taking longer now
    
    
• Another option is to make changes to the User/UserGroup resolution in EEM.  But it also means you need to make sure that your user authorization requirements match the needs. So, test this to make sure and/or make adjustments to your current architecture to make sure users are not losing authorizations
  • Modify the Group Resolution Level in EEM to "Resolve direct groups only."
  • Log into the Embedded Entitlements Manager (EEM) UI as eiamadmin
  • Navigate to the Configure tab.
  • Select User Store in the top menu, then click User Store in the left-hand pane.
  • Select Group Configuration and then Settings.
  • Locate the Global Group Configuration section.
  • Change the Group Resolution Level field to Resolve direct groups.
    Note: This ensures EEM only identifies groups the user belongs to directly, significantly reducing authorization time.
  • Click Save.
    Note: Repeat these steps on BOTH EEM servers if configured in a High Availability (HA) pair.
    
    

 

Additional Information

If more granular EEM tracing needs to be done, refer to this article.