Nolio Release Automation Pre-Upgrade Review Call Questions

Products

CA Release Automation - DataManagement Server (Nolio) CA Release Automation - Release Operations Center (Nolio)

Issue/Introduction

As a Nolio Release Automation customer planning on upgrading Nolio, what are the best practices in ensuring a successful upgrade?

The Nolio Release Automation Technical Documentation landing site provides useful links that cover Release Notes, Installing, Upgrading, Securing, Maintaining and Administering Nolio. The Resolution and Additional Information sections of this article include best practices to help guide you in developing an upgrade plan that Support can review with you.

A couple weeks ahead of an upgrade, please:

Open a case with Support. You'll be asked for an upgrade plan. This article can be used as a basis for your upgrade plan. This is based on upgrade plans Support has seen in the past that have the most success. An upgrade plan is typically a couple of pages long, but sometimes can be detailed down to the smallest detail so that nothing is missed and Support knows what's happening if something goes wrong and troubleshooting is needed.
Contact your account management team and request a Hot Site for your upgrade event so we can engage you and perform a pre-event review with you to ensure everything goes smoothly

Resolution

Pre-Event Planning

Testing:

Did you test this in non-production first?
- If possible, configure your Test Environment to mimic the Production Environment.
- If test and production are not similar, what is difference between the two?
Create a Testing Plan to include:
- Tests to be used on Test Environment to validate upgrade
- Tests to be used on Production Environment to check if upgrade is successful and doesn't need to be rolled back.
- Tests to comprehensively validate that all deployments are working as planned once the upgrade has been fully completed.
Did you run into problems with the non-prod upgrade? If so, what sorts of problems did you run into? The workarounds/solutions should be noted.

Backup plan:

What will you be backing up prior to upgrade?
When will you take the backup?

Backout/rollback plan:

When will you make the decision to back out in case something goes wrong?
What steps will be taken to back out of the upgrade?
Do you have a workaround in the meantime?

Timeline should include:

When does the installation or upgrade begin (date and time)?
What date/time will you stop components that need to be stopped?
For each component:
- What date/time will do you plan on updating each component?
- How long will these take?
- When do you expect to be finished with each component?
- How long will testing take to be sure it upgraded correctly?
When will you make the decision to back out in case something goes wrong?
When do you plan on having the upgrade end (date and time)?
If there is monitoring of Nolio components, ensure they are turned off or ignored during upgrade window.

Technical:

Components:
- What components are being upgraded?
  - Core components
    - Management server(s)
    - Execution server(s)
    - Repository server(s)
    - Agents
      - Send mail with Nolio/CARA Agent pre-requisite details to all Application teams and request them to validate that target servers meet the Pre-requisites.
      - Also, when upgrading agents, check with all Application teams to ensure that agents are up and running during the upgrade window.
  - Other
    - Are there any 3rd party components being upgrade? Examples:
      - Database (from, to type and versions?)
      - OS patches (from, to flavor and versions?)
      - OS in general (from, to flavor and versions?)
- What steps/commands are you using to upgrade each component?
- Are you copying a database?
  - If so, is there enough space on the target server?
- Is this an upgrade of a current system?
- Are you migrating data from one system to another?
- Is there any pre-work to be done on the system before stopping to upgrade?
- In what order will you be stopping running components?
- In what order will you be starting components back up?
  - What commands will you use to start them up?
  - What user and environment settings are normally used to bring up the processes? What will be used during and after the upgrade?

If something goes wrong:

What is an acceptable situation/workaround if something does go wrong? How long can it stay like that?
What is your backout plan for each component?
Who from your team needs to be available for troubleshooting? This should include:
- All key personnel/stakeholders within your organization (Operations, Database Management, Networking, Release Managers, etc..).
- CA/Broadcom Technical Support. Do you have the support number to call in case something goes wrong? Severity 1 cases cannot be raised from the website. Use a new case when running into a new issue.

Additional Information

Upgrade Planning

The guidance offered in the Upgrade Nolio Release Automation page should be used as a basis for your upgrade plan. This section is split into two parts to:

offer additional guidance (not explicitly documented in the Upgrade Nolio Release Automation page).
expand on points raised in the documentation.

Additional guidance (not explicitly documented in the Upgrade Nolio Release Automation page):

Review the System Requirements page to make sure the versions of Operating Systems and Databases used in your environment is supported by this version of Nolio.
If your environment uses High Availability for the datamanagement component, be sure to follow the instructions outlined here:
- Upgrade a High Availability Deployment
- Apply Patches to a High Availability Installation
The recommended goal should be to upgrade the latest supported version of Nolio and then apply the latest cumulative patch.
- Cumulative fixes can be found here: CA Release Automation Solutions & Patches
  Note: For Nolio v6.7, b398 (aka 6.7.3) cumulative fix, at a minimum, or above is recommended.
- Nolio 6.7 Release Notes for cumulative fixes can be found here: Release Notes
  Note: Please review the Release Notes for the cumulative patch you are applying and the Release Notes for cumulative patches that precede it. For example, if you're applying 6.7.6 and only read the release notes for it then you would not learn/plan for the Known Issue Nolio Agents Unreachable After 6.7.0.b398 - unless you read the Release Notes for 6.7.3.
Check to see if the "Run As User" functionality is using sudo. If so, verify configuration files won't be overwritten during the agent upgrade or backup and then restore after upgrade.
Review any custom Action Packs or RDK Action Packs prior to upgrading.
Run CA Release Automation Database Care and Feeding scripts post upgrade to clean old entries.
Take an inventory of Plugins (ex: Jenkins, TeamCity, TFS, etc.) and determine if updates are necessary.

Expand on points raised in the documentation:

Backup the jre/lib/security/cacerts keystore on agent machines that communicate with https servers before upgrading the agents to 6.6. In order for an agent to establish a successful SSL Handshake with https servers (via a variety of actions) the agent needs to have that https servers certificate imported into its cacerts keystore. Having a backup of this keystore will ensure that you can restore the needed certificates. This is related to the following KB article: agent certificate problems after upgrade
As noted in the Upgrade Nolio Release Automation, it is recommended to backup up the file system of management and execution servers. A full backup of the installation folder will help ensure that nothing important is lost such as:
- Action pack files
- Configuration files that have been modified (ex: server.xml, log4j.properties, cacerts, etc..)