vRealize Automation upgrade fails due to duplicates in the vRealize Orchestrator database

Products

• VMware Aria Suite

Issue/Introduction

• When upgrading vRealize Automation appliances, you may experience failures related to duplicates in the database for the vRealize Orchestrator service.
   
• In the vRealize Automation VAMI UI, you see the failure messages similar to :
  Failed to install updates(Error while running post-install scripots)
  VA-check: finished
  Pre-install: finished
  Post-install: failed
  Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.
   
• In the /var/log/bootstrap/postupdate.log file, you see the entries similar to:

Orchestrator Configuration Tool. Version: 7.4.0.8071781 Build: 8071781
 
Can not read the Orchestrator pid from /var/lib/vco/app-server/logs/tomcat.pid
Start configuring database settings.
Configuring database settings finished successfully.
Start updating database scheme.
ch.dunes.vso.db.check.CheckDuplicatesException:
Duplicate entries found in the Orchestrator database:
 
System tables duplicates:
        2 items with ID '<UUID>' in database table 'vmo_scriptmodulecontent'
.
.

Resolve duplicates by deleting unnecessary items.
java.lang.RuntimeException: ch.dunes.vso.db.check.CheckDuplicatesException:
Duplicate entries found in the Orchestrator database:

Resource element duplicates:
1 item with ID '<UUID>' and name 'ko.properties'
1 item with ID '<UUID>' and name 'fr_FR.properties'
1 item with ID '<UUID>' and name 'zh_CN.properties'
(and many more)

vRealize Automation 7.5+:

Duplicates found in the embedded vRealize Orchestrator database:
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=InstanceDetails
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=General
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=Allocation
Script /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates failed, error status 1

Environment

VMware vRealize Automation 7.x
VMware vRealize Automation 6.2.x

Resolution

VMware is aware of this issue.

A resolution for this issue is being evaluated for inclusion in a future release.

To work around this issue when vRO is using the embedded vRA psql database, perform the following steps:

For vRealize Automation 7.0 - 7.4:

1. Revert vRealize Automation environment back to pre-upgrade snapshots including IaaS components and SQL Database.
2. Ensure you have proper backups of the environment prior to proceeding!
3. Download the attached zip file and extract the contents.
4. Copy the vco-cfg-cli-7.4.0.23640-8992942.noarch.rpm  file to a directory on your vRealize Automation appliance.
5. Install the package using the command: 

   • rpm -hiv vco-cfg-cli-7.4.0.23640-8992942.noarch.rpm

6. Run the command:  

   • /usr/lib/vco-cli/bin/vro-configure.sh db-validate

7. Copy the list of duplicate entries (focused on id and name) for reference.
8. Connect to the postgres database using the command:
   • su postgres
   • /opt/vmware/vpostgres/current/bin/psql vcac
9. Run the following command(s) by referencing the correct ID and NAME from step 7 to remove one of each of the duplicate entries (it doesn't matter which is removed): 
   • delete from vmo_resourceelement where categoryid = 'cbd45656-bfb7-3275-8c1c-7ab8aa6ee487' and name = 'de_DE.properties'; ​
   • delete from vmo_configelement where categoryid = 'cbd45656-bfb7-3275-8c1c-7ab8aa6ee487' and name = 'de_DE.properties';
10. Exit the postgres command using: \q
11. Re-run the command to ensure you have no further duplicates:  

    • /usr/lib/vco-cli/bin/vro-configure.sh db-validate

12. The correct response should be: 

    • The command does not need a database configuration update.

13. Create additional backups/snapshots and retry your vRealize Automation upgrade.

For vRealize Automation 7.5+:

In 7.5 and 7.6 the upgrade automatically checks for duplicates and reports on those tables that are duplicates. Once duplicates are found you need to delete the duplicates. You may run the below command to check after deletion that all tables are removed.

1. Check for duplicate elements in versions 7.5+:

cd /usr/lib/vco/tools/configuration-cli/bin/
command : ./vro-configure.sh db-validate

2. If there are no duplicates you should see something to the below:

Orchestrator's root folder: /var/lib/vco
Orchestrator Configuration Tool. Version: 7.6.0.12923317 Build: 12923317
ValidateDbOptions{dbUsername='vmware', dbPassword='****', jdbcUrl='jdbc:postgresql://localhost:5433/vmware?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt'}
The command does not need database configuration update.

3. If duplicates are found there should be three primary types: Plugins, Version History, and Configuration Elements.  See below for instructions for each:
   1. Plugins are flagged as duplicate:
      1. Login to Orchestrator Client as Design user.
      2. Enter the Name output in the output in the NAME field output by upgrade.
      3. Open each duplicate and make note of the one with the most relevant content that you wish to keep.
      4. Right Click on the one you wish to remove and delete.
   2. Version history is flagged as duplicate:
      1. Login to the Orchestrator Client
      2. Resolve duplicates by deleting unnecessary items

Note: The auto-creation of an item means there is a running vRO process connected to your database. You must stop it and then delete the duplicate entry. The missing version history items are not critical and even they miss in different situations so it is not that important which one gets deleted.

Example:
Duplicates found in the embedded vRealize Orchestrator database:

- 1 items in table vmo_versionhistorycontent: ID= NAME=1.0.800
- 1 items in table vmo_versionhistorycontent: ID= NAME=-1.0

3. Configuration elements are flagged as duplicate:
   1. Login to the Orchestrator Client
   2. Search for the GUID
   3. Remove the duplicate element.

Example:
Duplicates found in the embedded vRealize Orchestrator database:
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=InstanceDetails
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=General
- 1 items in table vmo_configelement: ID=8a81a5a35f107593015f7871f8da0431 NAME=Allocation
Script /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates failed, error status 1

4. Corrupted, duplicate workflow tokens:
   1. Download the attached workflow package: com.vmware.o11n.bug_2444547.package
   2. Import the package
   3. Setup the /storage/db/pgdata/pg_hba.conf file to allow local connections from source IP:
      1. Under heading: # IPv4 local connections:
      2. Add the IP address of the vRealize Orchestrator node that will need access to the postgres database:
         Example:

host all all 192.168.110.75/24 md5

3. Restart vpostgres:

service vpostgres restart

4. Extract PSQL password and decrypt:

cat /etc/vcac/server.xml |grep password="*"

/usr/lib/vcac/tools/config/bin/vcac-config prop-util -d --p s2enc~646fCCJWPaSYJe91XKBpkrc1E+3N83n7ZIF3a+PGEaI= |less

Example password Output:
gRhoeG3+76n2o*nS​​​​​​

4. Create a new SQL Plugin host connection within the vRealize Orchestrator client by running the workflow named: "Library > SQL > Configuration > Add a database"

• Name: "Local vPostgres Instance"
• Database Type: postgreSql
• Connection URL: jdbc:postgresql://vra-01.vcloud.local:5432/vcac
• Select "Next"
• Session Mode: "Shared Session"
• User name:  vcac
• Provide decrypted credentials, example: gRhoeG3+76n2o*nS

4. Execute the workflow: "Delete Corrupted WorkflowTokens"

Note:  If the workflow fails, comment out: // System.log(token.state);
from within the Scriptable task object within the workflow "Delete Corrupted WorkflowTokens"

5. Reattempt the upgrade.