vRealize Operations Manager Application upgrade fails with the error “run sql db upgrade” failed

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The OS portion of the vRealize Operations Manager upgrade completes successfully.
• When attempting to apply the vRealize Operations Manager application PAK file, the upgrade fails with this message:

Installation failed with error FailedPAK action “run sql db upgrade” failed

• The /storage/vcops/log/sqldbupgrade.log contains entries similar to:

2019-09-05 11:35:38,862 ERROR [main]
com.vmware.statsplatform.persistence.util.sql.UpgradeActionExecutorManager
execute - Upgrade failed
 java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.RuntimeException:
org.postgresql.util.PSQLException: ERROR: right sibling's left-link doesn't
match: block 432 links to 6065 instead of expected 265 in index
"idx_alarm_resource_canceltime"

  at
com.vmware.statsplatform.persistence.util.sql.UpgradeActionExecutorManager
upgradeDatabase(UpgradeActionExecutorManager.java:201)

  at
com.vmware.statsplatform.persistence.util.sql.UpgradeActionExecutorManager
execute(UpgradeActionExecutorManager.java:70)

  at
com.vmware.statsplatform.persistence.util.sql.SqlDbUpgrade.main(SqlDbUpgra
e.java:49)

 Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: right
sibling's left-link doesn't match: block 432 links to 6065 instead of
expected 265 in index "idx_alarm_resource_canceltime"

  at
com.vmware.statsplatform.persistence.util.sql.UpgradeActionExecutorManager
executeSteps(UpgradeActionExecutorManager.java:226)

  at
com.vmware.statsplatform.persistence.util.sql.UpgradeActionExecutorManager
upgradeDatabase(UpgradeActionExecutorManager.java:197)

Environment

VMware vRealize Operations Manager 7.0.x
VMware vRealize Operations Manager 6.7.x
VMware vRealize Operations Manager 6.6.x

Cause

The issue can be caused by corruption on the disk or SEsparse.

Resolution

To resolve this issue, run a re-index on the alarms table in the postgres database.

1. Log into the vRealize Operations Manager Primary node as root vi SSH or Console.
2. Run the following command to reindex the alarm table:

su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -c 'reindex table alarm'"

Note: The prompt will display REINDEX once the reindex is completed.

3. Run \q to exit the database.
4. Repeat steps 1-3 on all other vRealize Operations Manager Analytic nodes in the cluster.

With the re-index successful, take new snapshots of the vRealize Operations Manager nodes and attempt the upgrade process again. It should now complete successfully.

Additional Information

For more information on SRsparse, see Virtual Machines running on an SEsparse snapshot may report guest data inconsistencies.
For more information on taking vRealize Operations Manager snapshots, see How to take a Snapshot of vRealize Operations Manager 6.x and later.