Identifying and removing blocking processes

Products

• CA Release Automation - Release Operations Center (Nolio)
• CA Release Automation - DataManagement Server (Nolio)

Issue/Introduction

Often time when we are running a process we get a message like the job is blocked waiting for previous process to finish. The problem accelerate when the system is used by many users and have lot of agents making it difficult to find which job is blocking.

The above problem is due to the fact that each agent participating or running a process can either run it exclusively (i.e. one after another in queue manner) or non-exclusively (running jobs in parallel). During the design of process in RA, the designer creates various server types in application and assign these agents under those server types, where each server type properties control exclusiveness mentioned prior.

Environment

Release Automation: 6.x

Resolution

To address above issue we first need to identify which jobs are blocking other jobs. There are various way to overcome this situation listed below.

1. If you have very few application and not large agents sets.
   
   • Steps to unblock jobs one by one using ROC UI
     
     1. Login to ROC and navigate to Environments-> Execution overview
     2. Filter the process run, make sure to uncheck all project option making it as "None". An example process list will be shown

        

     3. Navigate to the process which is "In Queue", identify server participating in that run and stop/remove running process which are halted or running on those agents for long time.
   • Removing jobs via JMX: Please refer to document Terminate a Process That Is Not Responding in CA Release Automation - Article ID: 35856     
2. If you have many application and large number of agents participating across various server types
   
   • This case is complex where you have many agents participating in various server types in various application and hard to track them manually one by one. Please find below process to narrow down jobs identification and removal as per below process.
     
     • Login to JMX console of NAC, ideally it will be at http://<NAC-HOST>:<JMX-PORT> for example http://localhost:20203
     • Invoke listBlockedJobs at http://<NAC-HOST>:<JMX-PORT>/invoke?operation=listBlockedJobs&objectname=noliocenter%3Atype%3Dinfo which will give us the list of job id's of all processes which are blocked.

• Now we need to invoke listActiveRuns at http://<NAC-HOST>:<JMX-PORT>/invoke?operation=listActiveRuns&objectname=noliocenter%3Atype%3Dinfo the name of all the jobs which are active running and need to get the job name of job id listed above.

 

• Once we have job name and jobs id, which are been blocked we will now get the list of agents on which these jobs are getting blocked. Invoke method listAgentQueues at http://<NAC-HOST>:<JMX-PORT>/invoke?operation=listAgentQueues&objectname=noliocenter%3Atype%3Dinfo it will give us list of all agents queue.

 

• Now once we have agent queue, we know which process on which particular agent is blocked by which running process. For example from above screen shot I can see that I have two agents in my system i.e. agent [1] and agent [3] and there is no running process on agent [1] but 3 processes on agent [3] out of which 2 process are in queue waiting for running process with job-id 70 to finish. Now knowing the job name I can just go ahead and remove those processes either via JMX or via ROC UI