VMware Smart Assurance: Unable to login to NCM GUI after upgrade to version 10.1.4

Products

• VMware

Issue/Introduction

Symptoms:

After successful upgrade to NCM 10.1.4, unable to login to GUI with message: org.postgresql.util.PSQLException: ERROR: relation "cm_cel_audit_record" does not exist.
Logs (powerup.log/server.log) on NCM Application server at $VOYENCE_HOME/ncmcore/logs have same error message.

Installation logs at $VOYENCE_HOME/logs directory have below messages:
Continuing with the Upgrade...
Redirecting to /bin/systemctl status controldb.service
psql: warning: extra command-line argument "-c" ignored
psql: warning: extra command-line argument "alter table voyence.cm_cel_audit_record alter column arrival_time set Default now();" ignored
psql: warning: extra command-line argument "-c" ignored
psql: warning: extra command-line argument "alter table archive.cm_cel_audit_record alter column arrival_time set Default now();" ignored

(or)

NCM setup upgrade to 10.1.4 version is stuck during Database upgrade.

Environment

VMware Smart Assurance - NCM

Resolution

As of July 2021, this is identified as a known issue, occurring/reproducible only in few environments during upgrade to NCM 10.1.4 and will be fixed in NCM upcoming release (version 10.1.6).

Workaround:

As a workaround, follow the below instructions:

First Step -
If there is a VM snapshot captured prior to NCM 10.1.4 upgrade on the VM, rollback to the snapshot.

(or)

If there is NCM data backup file (Ex: backup-image-10.1.3.0.37.CS.Jul-28-2021-02.00.tgz) captured prior to upgrade activity (generally available at $VOYENCE_HOME/data-image directory, as NCM by default captures daily backup):
a) Place the backup file at a different directory other than $VOYENCE_HOME on the server. 
b) Uninstall NCM.
Steps are available at https://docs.vmware.com/en/VMware-Smart-Assurance/10.1.4/ncm-installation-guide-1014/GUID-2E733EAB-5A41-4356-9A67-D0C82F8293F4.html
c) Install NCM prior version on the server from which 10.1.4 upgrade was performed. 
d) Restore the backup image (from step a) with following commands:
source /etc/voyence.conf
cd $VOYENCE_HOME/tools
perl restore.pl <Absolute path to backup image file/backup-image-.*.tgz>

 

Second Step -
Once either of the option in first step is performed, download the attached zip file from the KB & follow the below instructions.

1. Unzip the NETWORK_CONFIGURATION_MANAGER_10_1_4_0_Linux.zip to extract install.sh and database-changes.pl files.
2. Go to the location where NCM 10.1.4 GA installer binaries (downloaded from VMware Customer Connect portal) are extracted.
3. Take the backup of the existing install.sh and Utils/database-changes.pl files
4. Replace the existing install.sh file with the newly extracted install.sh file.
5. Replace the existing Utils/database-changes.pl with the newly extracted database-changes.pl file.

Additional Information

Attachment on KB -
File Name: NETWORK_CONFIGURATION_MANAGER_10_1_4_0_Linux.zip
Zip File Sha256sum: aed62fdcf3cfaf916920431641f3326341fcda7674ca667150435d4233411594