Data Repository upgrade fails due to missing libcrypto.so.1.1 file

Products

• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

Upgrading the DX NetOps Performance Management (PM) Data Repository (DR) Vertica database cluster to new releases with Vertica 10.1. This began with r21.2.3 PM versions.

 

Upgrading Vertica to 10.1 fails

 

The cluster runs on RH 7.x and has FIPS enabled.

 

The dr_install.sh runs and fails. Checking the log it creates in /opt/CA/IMDataRepository_vertica10/log shows all nodes were successful updating to the 10.1 Vertica rpm.

 

After that it shows the install_vertica run fails when it can't find a library file.

===============================================================================
/opt/vertica/sbin/install_vertica_verify_libraries: /usr/lib64/libcrypto.so.1.1 does not exist! Expecting to find this library since this is a FIPS system
/opt/vertica/sbin/install_vertica: Cannot verify shared libraries
install_vertica script failed with exit code 1..........................[FAIL]
===============================================================================

We can't install the openssl1.1 package to get the missing file. It's only supported on RH 8.x and these servers are RH 7.x that run openssl1.0, the newest/last version supported on RH 7.x.

Environment

DX NetOps Performance Management releases r21.2.3+ where Vertica 10.1 upgrades begin.

Cause

Vertica 10.1 only supports FIPS on RH 8.x systems.

Resolution

An upgrade can be performed to Vertica 10.1 in this scenario using special steps. This path ensures we maintain the FIPS certification required.

The upgrade path for this scenarios is as follows.

1. Create a full backup of the database following the steps found in the Back Up the Data Repository documentation.
2. Stop the database. The Restart the Data Repository following documentation covers stopping and starting the database.
3. Upgrade the OS from 7.x to 8.x. Note that RedHat appears to recommend going to the latest RH 7.x before going to RH 8.x.
4. Enable FIPS:
   • If upgrading in place from RH 7.x to RH 8.x ensure FIPS remains enabled post upgrade.
   • If performing a clean new install of RH 8.x ensure FIPS is enabled post install.
5. Upgrade Vertica
   1. If upgrading Vertica on a systems that was upgraded in place from RH 7.x to RH 8.x follow the Upgrade the Data Repository documentation.
   2. If performing a new Vertica install on a system that had a clean new RH 8.x install performed follow the Install the Data Repository (new install) documentation.
6. Start the database (for in-place OS upgrade to RH 8.x) or restore then start the database (new install of RH 8.x). 
   1. If the OS is a fresh clean new RH  8.x install we'll need to:
      1. First restore the database following the Restore the Data Repository documentation.
      2. After the restore, start the DB which will complete the upgrade to Vertica 10.1.1.
          
   2. If the OS was upgraded in place from RH 7.x to RH 8.x:
      1. No DB restore is required.
      2. Start the DB to complete the DB upgrade to 10.1.1.

Additional Information

This is what Vertica refers to as a "Nonsequential FIPS Database Upgrade".

Under normal circumstances they recommend sequential upgrades. IE going from 9.1->9.2->10.0->10.1 without skipping releases.

Performance Management takes advantage of Nonsequential upgrade paths to help minimize upgrade down time and complexities.

Additional information from Vertica about these types of upgrades is found in their Nonsequential FIPS Database Upgrade documentation.

https://www.vertica.com/docs/10.1.x/HTML/Content/Authoring/InstallationGuide/Upgrade/FIPSNonsequentialUpgrades.htm in order to verify the steps.