"DB schema hash mismatch File" error while upgrading vCenter server and fails with error 1603

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Windows vCenter Server upgrade fails with 1603 error message​​​​​​.
• You will see similar log entries as mentioned below

vminst.log
2019-11-05 21:29:48.790-07:00| vcsInstUtil-9451876| I: Leaving function: VM_FinishInstallWithError
2019-11-05 21:46:46.682-07:00| vcsInstUtil-13010631| I: PitCA_MessageBox: Displaying message: "Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details."
2019-11-05 21:48:53.135-07:00| vcsInstUtil-13010631| I: Entering function: VM_FinishInstallWithError
 
updateboot.log
2019-11-06T04:46:15.57Z ERROR updateboot Failed to start all services. RC=1. Details:^M
2019-11-06T04:46:15.041Z  Error waiting for service VMwareIdentityMgmtService start. Exit code: 1066 Current State: 1^M^M
Error executing start on service VMwareIdentityMgmtService. Details {^M^M

vpxd.log
2019-11-05T18:29:40.583-07:00 info vpxd[02648] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-11-05T18:29:40.583-07:00 info vpxd[06664] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-05T18:29:40.583-07:00 info vpxd[06664] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-11-05T18:29:41.201-07:00 info vpxd[16376] [Originator@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB
2019-11-05T18:29:41.403-07:00 info vpxd[10448] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 117, idle: 1
2019-11-05T18:29:41.403-07:00 info vpxd[03100] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-05T18:29:41.403-07:00 info vpxd[03100] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-11-05T18:29:42.232-07:00 info vpxd[16376] [Originator@6876 sub=vpxdVdb] DB Version set in vpxd.cfg = Microsoft SQL Server 2012 (SP4-GDR) (KB4057116)
2019-11-05T18:29:42.321-07:00 info vpxd[16376] [Originator@6876 sub=DbSchemaHash] [VpxdDbSchemaHash::GetSignatureFilename]: sqlDir: C:\Program Files\VMware\vCenter Server\vpxd\sql\
2019-11-05T18:29:42.321-07:00 info vpxd[16376] [Originator@6876 sub=DbSchemaHash] [VpxdDbSchemaHash::CheckSchema]: signature file: C:\Program Files\VMware\vCenter Server\vpxd\sql\signature_mssql.txt
2019-11-05T18:29:42.346-07:00 error vpxd[16376] [Originator@6876 sub=DbSchemaHash] [VpxdDbSchemaHash::CheckSchema] DB schema hash mismatch File: 4aec34f21815e498c3c79c3a68a32c3c DB: aaceaee476f7596046bb7ec21a8feaf9
2019-11-05T18:29:42.346-07:00 error vpxd[16376] [Originator@6876 sub=Main] Init failed. SystemError: class Vmomi::Fault::SystemError::Exception(Fault cause:
vmodl.fault.SystemError
--> )
--> [context]zKq7AVMEAAAAADV5xgANdnB4ZAAA6qcedm1hY29yZS5kbGwAAKANBgAONQYAOj8CAfVNAHZtb21pLmRsbAABmTUBAkzuIXZweGQuZXhlAALRMyICK6JEAleqRIKrtgoBA/ITAEtFUk5FTDMy
LkRMTAAE9FQBbnRkbGwuZGxsAA==[/context]
2019-11-05T18:29:42.348-07:00 error vpxd[16376] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-11-05T18:29:42.348-07:00 info vpxd[16376] [Originator@6876 sub=SupportMgr] Wrote uptime information19078368711

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x

Cause

This issue occurs in one of the cases:
 

• The vCenter Server database was restored with an earlier version that is not compatible with the current running version of vCenter Server, resulting in a database schema mismatch.
• During an upgrade of vCenter Server, the database upgrade process failed after modifying the current database schema.
• User ID defined in vCenter database and vpxd.cfg file are different

Resolution

To verify if there is a vCenter Server database schema mismatch:
 

1. Connect to the vCenter Server database.
   • SQL - Use SQL Management Studio to connect to vCenter Database.
   • Postgres - Run the below query to connect to VCDB
     • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
2. Run below query to list the rows in the VPX_Version table in the vCenter Server database:
   
      Select * from VPX_Version;
3. Note the value in the VERSION_VALUE field.
4. Open the vpxd.log file and locate the initial header of the file. The header shows the current version of vCenter Server.
   
   For example, the header appears similar to:
   Section for VMware VirtualCenter, pid=4384, version=6.5.0, build=9451637, option=Release
5. Compare the vCenter Server versions noted in Step 3 and Step 4. If the versions do not match, the vCenter Server database is running an incorrect schema.

    6. To upgrade the database schema to the correct version:

• Uninstall vCenter Server.
• Restore the vCenter Server database from a backup.
• Reinstall vCenter Server using the "Use existing database" option.

    7. If the versions match, then validate the database_user_id used to login to System DSN (vCenter Database) and match it with the user_id defined in vpxd.cfg file


To change the vCenter Server user ID in vpxd.cfg file, perform the following:
 

1. Stop vpxd service.
2. Navigate to this location,
   • Windows: C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx 
   • VCSA: /etc/vmware-vpx/
3. Take a backup of vpxd.cfg
4. Open the vpxd.cfg file in a text editor.
5. Locate the <DB> element and modify the value for <key_2> to reflect the new database user ID.
   ...
   <DB>
   <key_2>database_user_id</key2>
   <key_3></key_3>
   </DB>
   ... 
6. If you are using Windows Integrated Authentication to connect to the vCenter Server database, remove the value for <key_3>.
7. Save the vpxd.cfg file.
8. Click Start, right-click Command Prompt, and select Run as administrator to open a command prompt as an administrator. If its VCSA go to Step 10
9. Navigate to C:\Program Files\VMware\vCenter Server\vpxd
10. Run command vpxd -p and update the same DB password twice.
11. Start vpxd service.
12. Re-run the installer to perform the upgrade.