Fault Tolerant Data Aggregator upgrade shows a fatal error

Products

• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

We are currently operating on version 22.2.10 and are in the process of upgrading to version 22.2.11. This is a fault-tolerant (FT) setup.

Upgrade Steps:

1. NetOps Portal Upgrade

   • Action: Upgraded NetOps Portal to 22.2.11 successfully.

2. DA Proxy Upgrade

   • Action: Upgraded DA Proxy to 22.2.11 successfully.

3. DA Secondary Upgrade Preparation

   • Action: DA Primary is active, put DA Secondary on maintenance.
   • Upgrade: Upgraded DA Secondary to 22.2.11 successfully.
   • Action: Put DA Secondary on active, DA Primary on maintenance.

4. DA Primary Upgrade

   • Action: Upgraded DA Primary to 22.2.11.
   • Result: Not successful.

Installer Output

The installer ends at the CLI with the following message:

The installation of Data Aggregator is complete, but some errors occurred during the install. Please see the installation log for details. Press "Done" to quit the installer. PRESS <ENTER> TO EXIT THE INSTALLER:

 

Log Summary

The log shows the following summary:

Installation: Unsuccessful. 863 Successes 0 Warnings 0 NonFatalErrors 1 FatalErrors

 

Specific Error

The log details include this specific error:

Custom Action: com.ca.im.dm.core.installer.CustomCode.dbmigration.DBMigrationAction Status: FATAL ERROR Additional Notes: FATAL ERROR - class com.ca.im.dm.core.installer.CustomCode.dbmigration.DBMigrationAction FatalInstallException: com.zerog.ia.api.pub.NonfatalInstallException Failed to alter alarm tables: com.ca.im.dm.core.installer.CustomCode.DmInstallException unable to execute SQL statement cause: ROLLBACK: Column "metric_values_min" is already of type "varchar(255)"

Environment

All supported DX NetOps Performance Management releases

Upgrading 22.2.10 to 22.2.11

Cause

Normally, the first Fault Tolerant (FT) Data Aggregator (DA) updated is the one that updates the Disaster Recovery (DR) database schema with necessary changes. The second FT DA updates should recognize that the first DA has already completed these changes and should not attempt to execute them again. If it does try to make these changes, it is attempting to redo alterations that were already successfully completed by the first DA update.

In this scenario, it is likely that a network issue with the problematic DA and its access to the shared data directory caused this problem. The second DA may have incorrectly attempted to perform schema updates due to this network disruption.

Resolution

The error encountered during the upgrade of the Data Aggregator (DA) to version 22.2.11 is benign. The initial updates to the DA were completed successfully, and the Data Repository (DR) database schema has been updated correctly.

To resolve the issue, you can safely bring the DA that encountered the install error out of maintenance using the following command:

/opt/IMDataAggregator/scripts/dadaemon activate

This command should be run from the default directory where the DA is installed. After executing this command, the DA will be active and operational.