vRealize Operations 8.10.x upgrade fails at step 13 of 14
search cancel

vRealize Operations 8.10.x upgrade fails at step 13 of 14

book

Article ID: 343866

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • After attempting an upgrade to vRealize Operations 8.10.x, in the admin UI you see the upgrade failed at step 13 with the following error:
Failedresource key=pak_manager_action_failed, resource args=[run master postgres db upgrade]
  • The /storage/log/var/log/vmware/vcops/stats-post-upgrade_*.log and/or /storage/log/vcops/log/centralsqldbupgrade.log contain an error similar to:
15:16:54.092 [main] INFO   - upgrader is successful, now update db version to:273
15:16:54.095 [main] INFO   - Invoking upgrade() method of class: com.vmware.vcops.dbupgrade.postgres.centraldb.upgrade.EpopsObjectsMarkAsNotExistUpgrader
15:16:54.095 [main] INFO  com.vmware.vcops.dbupgrade.postgres.centraldb.upgrade.EpopsObjectsMarkAsNotExistUpgrader - EpopsObjectsMarkAsNotExistUpgrader start
15:16:54.104 [main] INFO  com.vmware.vcops.dbupgrade.postgres.centraldb.upgrade.EpopsObjectsMarkAsNotExistUpgrader - EpopsObjectsMarkAsNotExistUpgrader found %d objects 6
15:16:54.104 [main] ERROR  - error:
java.lang.NullPointerException: null
at com.vmware.vcops.dbupgrade.postgres.centraldb.upgrade.EpopsObjectsMarkAsNotExistUpgrader.upgrade(EpopsObjectsMarkAsNotExistUpgrader.java:43) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.processJAVA(CentralSqlDbUpgrade.java:206) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.upgradeControllerDB(CentralSqlDbUpgrade.java:261) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.main(CentralSqlDbUpgrade.java:356) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
15:16:54.105 [main] ERROR  - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.processJAVA(CentralSqlDbUpgrade.java:211) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.upgradeControllerDB(CentralSqlDbUpgrade.java:261) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.main(CentralSqlDbUpgrade.java:356) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
Caused by: java.lang.NullPointerException
at com.vmware.vcops.dbupgrade.postgres.centraldb.upgrade.EpopsObjectsMarkAsNotExistUpgrader.upgrade(EpopsObjectsMarkAsNotExistUpgrader.java:43) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
at com.vmware.vcops.dbupgrade.postgres.centraldb.CentralSqlDbUpgrade.processJAVA(CentralSqlDbUpgrade.java:206) ~[vcops-dbupgrade-1.0-SNAPSHOT.jar:?]
... 2 more
15:16:54.107 [main] INFO  com.vmware.statsplatform.persistence.SpringService - shutting down persistence spring application context


Environment

VMware vRealize Operations 8.3.x
VMware vRealize Operations 8.2.x
VMware vRealize Operations 8.4.x
VMware vRealize Operations 8.5.x
VMware vRealize Operations 8.6.x

Cause

This issue is caused by leftover End Point Operations adapter instances, End Point Operations objects or End Point Operations related child objects.

Resolution

To resolve this issue, remove any End Point Operations Adapters or End Point Operations objects.  Complete all of the steps below to ensure all End Point Operations related items are removed.

Note: If there are no End Point Operations Adapters, simply continue on to the steps to remove End Point Operations Objects.  

Remove End Point Operations Adapters

Utilize these steps to use the Swagger UI and remove the End Point Operations adapters via API.
  1. Navigate to https://primary-ip-or-fqdn/suite-api.
Note: Replace primary-ip-or-fqdn with the IP or FQDN of the vRealize Operations Primary node.

Example: https://192.168.3.10/suite-api
  1. Click Authorize and enter the local admin user credentials, then click Authorize.  Once the Authorized message appears, click Close.
  2. Expand Adapters.
  3. Expand GET /api/adapters.
  4. Click the Try it out button.
  5. Enter EP Ops Adapter in the adapterKindKey box.
  6. Click the Execute button.
  7. If any End Point Operations Adapters are found, the Response Body will contain entries with the following text: 
            "adapterKindKey": "EP Ops Adapter",
Note: If there are no End Point Operations Adapters found, skip to the Remove End Point Operations Objects steps.
  1. Note the IDs of any End Point Operations Adapters found by copying the value of the ID field.
Example:"id": "d1027663-5fb7-4ad2-a1ca-1801e5b874a5"
  1. Expand DELETE /api/adapters/{adapterId}.
  2. Click the Try it out button.
  3. Paste one ID noted in step 8 in the adapterId field.
  4. Set the deleteResources dropdown to true.
  5. Click the Execute button.
  6. Verify that the HTTP Response Code is 204.
  7. Repeat steps 12-15 for any other End Point Operations adapter instance IDs noted from step 8.
 

Remove End Point Operations Objects

Utilize these steps to remove the  End Point Operations objects.
  1. Log into the vRealize Operations Manager UI, at https://primary-ip-or-fqdn/, with the local admin user.
Note: Replace primary-ip-or-fqdn with the IP or FQDN of the vRealize Operations Primary node.

Example: https://192.168.3.10/
  1. Navigate to Environment > Inventory.
  2. Type EPOPs in the filter field on the right and press enter.
  3. Select any objects that has an object type or adapter type containing the term EPOPS or EP Ops, and press the Delete Object button (red X) in the toolbar.


You can now retry the upgrade to vRealize Operations Manager 8.10.x.

Additional Information

Impact/Risks:
Prior to performing the steps in this article, roll back vRealize Operations to snapshots taken prior to the failed/hung upgrade attempt.

Powered by Wolken Software