"Internal error occurs during VMware vCenter Server Database pre-upgrade checks" upgrading vCenter Server Appliance to 6.x/7.0

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server Appliance upgrade to 6.x/7.0 fails with the error:
  • Internal error occurs during VMware vCenter Server Database pre-upgrade checks
  • Cannot validate target appliance configuration as not enough information from the source %(0)s can be collected

• In the /var/log/vmware/upgrade/CollectRequirements_com.vmware.vcdb_YYYY_MM_DD_HH_MM.log file, you see entries similar to:

[YYYY-MM-DDTHH:MM:SS] INFO vcdb.const read_db_value:
[YYYY-MM-DDTHH:MM:SS] ERROR __main__ Upgrade Phase 'vcdb:CollectRequirements' failed. Exception: could not convert string to float:
Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/payload/componentPhaseLauncher.py", line 461, in main
    executionResult = systemExtension(exeContext)
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/libs/sdk/extensions.py", line 94, in __call__
    result = self.extension(*args)
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/libs/sdk/extensions.py", line 110, in _func
    return func(*args)
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/payload/component-scripts/vcdb/__init__.py", line 22, in do_collect_upgrade_requirements req = do_linux_collect_req(context)
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/payload/component-scripts/vcdb/vcdb_collect_requirements_functions.py", line 374, in do_linux_collect_reqsize_core, size_set, size_core_et = core_set_size_postgres()
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/payload/component-scripts/vcdb/vcdb_db_utils.py", line 279, in core_set_size_postgres
    size_core = num(get_psql_query_result(sql_core))
  File "/tmp/vmware-upgrade-temp-dirx7dSM8r1MC/tmpSKh6ZXVEIo/payload/component-scripts/vcdb/vcdb_db_utils.py", line 45, in num
    return float(str_num)
ValueError: could not convert string to float

• In the /var/log/vmware/upgrade/vcdb_req.out file, you see entries similar to:
  
  Password for user postgres:
  psql.bin: fe_sendauth: no password supplied

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x

Cause

This issue occurs when .pgpass file is not available on the source vCenter Server Appliance.

Resolution

To resolve this issue:

1. Check for .pgpass file under /root folder. If not available, proceed with below steps to recreate the file:
   1. Login to Source VCSA using SSH.
   2. Execute the command ls /root/ -larth to verify the file presence.

Note: Take Snapshot of Source vCenter Server Appliance before proceeding with below steps.

2. Copy PGUSER Password from the /etc/vmware-vpx/embedded_db.cfg file.

For example:
 

root@testvcenter [ / ]# cat /etc/vmware-vpx/embedded_db.cfg | grep PGUSER_PASSWORD

PGUSER_PASSWORD='PG-PASSWORD'

3. Create the .pgpass file under /root with the new password:
   • touch /root/.pgpass
   • chmod 0600 /root/.pgpass
   • vi /root/.pgpass
   • Press Insert Key and Copy below contents to the file, replace PASSWORD with the password copied from above step and press ESC Key and wq! to exit the VI editor:

localhost:5432:postgres:postgres:PASSWORD
127.0.0.1:5432:postgres:postgres:PASSWORD
localhost:5432:VCDB:postgres:PASSWORD
127.0.0.1:5432:VCDB:postgres:PASSWORD
/var/run/vpostgres:5432:*:postgres:PASSWORD

4. Take a copy of /storage/db/vpostgres/pg_hba.conf file:

cp /storage/db/vpostgres/pg_hba.conf /storage/db/vpostgres/pg_hba.conf_backup

5. Edit the /storage/db/vpostgres/pg_hba.conf file using VI editor and change md5 to trust for below values:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

6. Save and close the pg_hba.conf file.
7. Stop and Start vPostgres service for the changes to be effective:

service-control --stop vmware-vpostgres
service-control --start vmware-vpostgres

8. Connect to the database as a postgres user. It does not ask for a password because we have changed the method in pg_hba.conf file to trust:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

9. Change the postgres user password with the password copied from the first step (from file /etc/vmware-vpx/embedded_db.cfg):

ALTER USER postgres PASSWORD 'new_password';
\q (to exit from postgres)

10. Revert the changes made on pg_ha.conf file, edit /storage/db/vpostgres/pg_hba.conf using VI editor and change trust to md5 for below values:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

11. Save and close the pg_hba.conf file.
12. Stop and Start vPostgres service for the changes to be effective:

service-control --stop vmware-vpostgres
service-control --start vmware-vpostgres

13. Connect to the database as a postgres user to verify the new password. If password is set properly, it will not ask for a password as it will take the password from .pgpass file:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
\q (to exit from postgres)
 

14. Continue with the upgrade process.

Additional Information

- In the step 3, enter the PASSWORD field without single quotes.