Deployment Blocked by Another Deployment

Products

• CA Release Automation - DataManagement Server (Nolio)
• CA Release Automation - Release Operations Center (Nolio)

Issue/Introduction

This problem may manifest in one of two ways:

• A deployment hangs with a "blocked" status.
• A deployment runs indefinitely, showing no sign of progress or failure, and may block other deployments waiting for it to finish. 

Environment

Release: NOLNAC99000-6.1-Nolio-Automation Center
Component:

Cause

Server Types can be configured to: Require exclusive execution
This is a property of server types. If the checkbox is checked then each time a job/deployment is executed on an agent assigned to that server type it will block other jobs from running until the first job is complete.

For example: 
Application "ABC" with Server Type "ABC" has its "Requires exclusive execution" checkbox checked. AgentA is assigned to Server Type ABC and a job (deployment is named: deploytest1) is started with a simple step (delay action set to 10 minutes).

Another deployment job (named: deploytest2) is started before deploytest1 is finished. If AgentA is being used in deploytest2 then its step will be blocked until deploytest1 is finished. 

There is no problem if deploytest1 runs as expected. But if there are errors in deploytest1 and its process enters a state of failed-paused then deploytest2 will be blocked until the problems in deploytest1 are resolved. That might mean fixing whatever problems and then resuming the job or maybe even cancelling the job. 
 

Resolution

These methods can be used to identify running jobs and processes. The steps for using JMX also describes how you can cancel the process and/or deployment. But whether or not you should do that depends on your situation. It is recommended that you understand which deployment/process is the culprit before simply cancelling it unless you're sure that it will not impact another team/environment and needs to be cancelled. Processes can run independent of deployments so it is recommended you check Noliocenter first for running processes to identify processes that have problems (assuming it is being blocked due to another process that's having problems). Using the Releasecenter JMX view to remove a deployment is an extreme measure and should only be used when there are 0 processes shown in Noliocenter. Remember, this can happen when there is a job running on the same agent that you're job is trying to use. And the job/process that is blocking your job was configured specifically to prevent other jobs running at the same time - likely because they're concerned with conflicts that might occur (whether it be file conflicts, resource conflicts, etc..). 

If your deployment is being blocked by a process in another deployment then you can probably quickly identify the job by looking at the dashboard in the Release Operations Center (ROC) and look into how long the current process has been running, whether an agent you're using is assigned, etc.. The area on the dashboard to focus on would be running and/or long running jobs. 

If you don't have any running jobs or if none of the deployments are using the same agent(s) as your deployment then you can use JMX to find processes. If you need to use JMX to find processes then once a set of processes have been found it gives you the following information:

• Process Name
• Process tag
• Time started
• Status

This information can be used by taking note of the process names, going back into the ROC -> Environments -> Process Execution where you can search for the process name for your various applications. Then you can look into what's happening and either cancel the process in Process Execution once you've determined the cause. 

Noliocenter (running processes)

1. Navigate to JMX (http://<NAC hostname>:20203).
2. Login with username/password nolio/nolio.
3. Scroll down to the section "Domain: noliocenter".
4. Select "noliocenter:type=info".
5. Click the "Invoke" button to the right of "listActiveRuns".
6. Find the ID of the hung process.
7. Go back to the previous page.
8. In the field to the right of "removeProcessRun", enter the ID of the hung process.
9. Click the "Invoke" button on the same row (for "removeProcessRun").

Releasecenter (running deployments)

1. Navigate to JMX (http://<NAC hostname>:20203).
2. Login with username/password nolio/nolio.
3. Scroll down to the section "Domain: releasecenter".
4. Select "releasecenter:type=Info".
5. Click the "Invoke" button to the right of "getAllActiveReleases".
6. Find the ID of the hung process.
7. Go back to the previous page.
8. In the field to the right of "removeRelease", enter the ID of the hung process.
9. Click the "Invoke" button on the same row (for "removeRelease").

Additional Information

If you need to selectively go through jobs that are returned by the listActiveRuns JMX method from above and figure out what Release Deployment a jobId might be a part of then the following query may help to identify the Deployment's Release ID and Name:

select release_id, release_name from execution_jobs where id = '<job_id_from_listActiveRuns>'

With this information you should be able to review the deployment by finding it by name from the Dashboard Running, Long Running, or Error State widgets. Or, by accessing the release using the following URL - replacing the releaseID at the end of the URL with the ReleaseID identified by the SQL Query: http://nolioservername:8080/datamanagement/#/main/deployments/<release_id_value_from_query>