VSA NCM Installer during upgrade from 10.1.x generates an error message pg_dumpall: could not connect to database "template1

Products

• VMware Smart Assurance

Issue/Introduction

To provide the steps to resolve the issue

Symptoms:
Upgrade from 10.1.1 to 10.1.6 causes full data loss in controldb if the installation generates the following error and it allowed to continue.

The NCM installer when trying to upgrade the Database Server generates the following error message

Redirecting to /bin/systemctl status controldb.service
DBPW is Encrypted
psql: FATAL: password authentication failed for user "pgdba"
password retrieved from file "/root/.pgpass"
psql: FATAL: password authentication failed for user "pgdba"
password retrieved from file "/root/.pgpass"
*     Taking backup...
pg_dumpall: could not connect to database "template1: FATAL:  password authentication failed for user "pgdba"
ERROR: pg_dumpall failed to execute during backup ...

Environment

VMware Smart Assurance - SMARTS

Cause

NCM versions 10.1.x require a 15-character password also must meet minimum requirements 
In this case NCM 10.1.1 was an upgrade from 9.6 and still had an 8-character password and this generated the error in the installer.  The pgdba user requirement was not met for the template1 database that allows the pbdump to process and dump the database contents required for the database to be upgraded.

Resolution

Before upgrading the Database Server from the 10.1.1 version of NCM to a newer version:

Make sure you have 50% or more free file space where the database resides
Have a backup of the Database Server, or a snapshot to restore the database server to prevent data loss and have a roll-back method.  

Should you receive this error message from the installer as mentioned above.  Type in Quit at the next prompt to cancel the installation.

Follow all of the steps to update the passwords on all servers and create the bundle.p12 as documented in the following 
document link:  
Change NCM passwords (vmware.com)

Please Note:  NCM 10.1.x has the following password requirements:

The new password must contain at least:
One letter in upper case
One letter in lower case
One special character
One numerical
Minimum fifteen characters

The special characters that can be used for the database password are:
Tilde (~)
Exclamation (!)
At (@)
Number (#)
Percent (%)

Additional Information

Impact/Risks:
High:  The database contents will be cleared if the installer is allowed to continue. 

Quit the installer to prevent data loss.