Upgrading Aria Operations Manager hangs. The process is stuck at step 10 of 14 "Applied product update" due to CustomRestPlugin.

Products

• VMware Aria Operations (formerly vRealize Operations) 8.x
• VMware Aria Suite

Issue/Introduction

• Aria Operations upgrade process gets stuck at step 10 out of 14 - "Applied product update" and it does not proceed from there.
• The build number in the admin UI gets updated to the new version but it is stuck in "Waiting for analytics" state and Cluster status is seen as 'Going online'.
• Similar log snippet is seen in analytics.log:
  2024-xx-xxx09:17:03,xx000 ERROR [Analytics Main Thread] com.integrien.analytics.AnalyticsMain.uncaughtException - Thread Analytics Main Thread threw an uncaught exception. Exception was: java.lang.NoClassDefFoundError: com/integrien/analytics/plugins/alertplugins/AlertPluginBase
java.lang.NoClassDefFoundError: com/integrien/analytics/plugins/alertplugins/AlertPluginBase
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
at java.lang.ClassLoader.defineClass(Unknown Source) ~[?:?]
................................
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at com.integrien.alive.common.plugins.generic.GenericPluginLoader.loadPlugins(GenericPluginLoader.java:147) ~[alive_platform.jar:?]
at com.integrien.analytics.plugins.TransmissionWrapper.loadPlugins(TransmissionWrapper.java:84) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
at com.integrien.analytics.AnalyticsMain.doRun(AnalyticsMain.java:598) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
at com.integrien.analytics.AnalyticsMain.run(AnalyticsMain.java:2020) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
Caused by: java.lang.ClassNotFoundException: com.integrien.analytics.plugins.alertplugins.AlertPluginBase
at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
... 14 more
2024-08-30T09:17:03,906+0000 ERROR [Analytics Main Thread] com.integrien.analytics.AnalyticsMain.uncaughtException - Terminating process with exit code: -1
2024-08-30T09:17:03,906+0000 INFO [Analytics Main Thread] com.vmware.vcops.platform.common.PlatformEnvironment.exitSystem - Terminating process with exit code -1 ...
2024-08-30T09:17:03,907+0000 INFORMATION [SystemExitThread] com.vmware.vcops.platform.common.PlatformEnvironment.run - exitSystem has been called by:
java.lang.Throwable: java.lang.NoClassDefFoundError: com/integrien/analytics/plugins/alertplugins/AlertPluginBase
at com.vmware.vcops.platform.common.PlatformEnvironment.exitSystem(PlatformEnvironment.java:260) ~[alive_platform.jar:?]
at com.vmware.vcops.platform.common.PlatformEnvironment.exitSystem(PlatformEnvironment.java:283) ~[alive_platform.jar:?]
at com.integrien.analytics.AnalyticsMain$3.uncaughtException(AnalyticsMain.java:1998) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
.........................
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at com.integrien.alive.common.plugins.generic.GenericPluginLoader.loadPlugins(GenericPluginLoader.java:147) ~[alive_platform.jar:?]
at com.integrien.analytics.plugins.TransmissionWrapper.loadPlugins(TransmissionWrapper.java:84) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
at com.integrien.analytics.AnalyticsMain.doRun(AnalyticsMain.java:598) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]
at com.integrien.analytics.AnalyticsMain.run(AnalyticsMain.java:2020) ~[vcops-analytics-1.0-SNAPSHOT.jar:?]

 

Environment

VMware Aria Operations

Cause

This issue occurs when an unsupported plug-in is installed. If there are any "CustomRestPlugin" being used, it leads to the upgrade failure.

 

Resolution

To resolve this issue you must remove the unsupported plugin prior to upgrade.  

To workaround this issue and uninstall the plug-in, please follow the steps below before proceeding with the upgrade:

1. Log in to the vRealize Operations Manager Master node as root through SSH or Console.
2. If the third-party plug-in is outbound, navigate to the directory:
   /usr/lib/vmware-vcops/user/plugins/outbound
   
   OR
   
   If the third-party plug-in in inbound, navigate to the directory:
   /usr/lib/vmware-vcops/user/plugins/inbound
   
   Note: Make a note of the name of the third-party plug-in you want to remove.
3. Delete the JAR file and directory associated with the third-party plug-in by running the following command:
   rm -r plug-in_name*
   
   Note: Replace plug-in_name with the name of the third-party plug-in found in step 2.
   
   For example:   rm -r rest-plugin*
4. Repeat steps 1-4 on all other nodes in the cluster.

After performing the above steps, upgrade can be attempted.