Aria Operations upgrade hangs at step 10 with analytics service failing due to JSON deserialization error

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

1. Analytics service fails to start during the course of upgrade, at stage 10/14.
2. Similar error stack observed in analytics-*.log:

ERROR [Analytics Main Thread]  com.integrien.analytics.AnalyticsMain.run - AnalyticsMain.run failed with error: AuthException: Auth Exception occurred : Auth Exception occurred : Auth Exception occurred : Parallel deserialization failure
com.vmware.vcops.auth.exception.AuthException: Auth Exception occurred : Auth Exception occurred : Auth Exception occurred : Parallel deserialization failure
        at com.vmware.vcops.auth.server.AuthorizationServer.initializeAuthObjects(AuthorizationServer.java:283) ~[vcops-auth-server-1.0-SNAPSHOT.jar:?]
...
Caused by: com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$UnknownFieldException: No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
---- Debugging information ----
message             : No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources
field               : allowAllResources
class               : com.vmware.vcops.auth.xml.bean.UserWrapper
required-type       : com.vmware.vcops.auth.xml.bean.UserWrapper
converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
line number         : -1
version             : 1.4.19
-------------------------------
        at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.handleUnknownField(AbstractReflectionConverter.java:520) ~[xstream-1.4.19.jar:1.4.19]

Environment

VMware vRealize Operations 8.10.x
VMware vRealize Operations 8.6.x

Cause

Aria Operations upgrades (specifically from 8.6.x to 8.10.x) failing due to JSON deserialization error occurring on auth UserWrapper rows having immediate allowAllResources child element.

Resolution

This issue is known to us for Aria Operations upgrades from version 8.6.x to 8.10.x. This might be fixed in the upcoming releases.

Workaround:

Follow the below steps:

1. Please check for error like "No such field com.vmware.vcops.auth.xml.bean.UserWrapper.allowAllResources" in the analytics logs and observe that the analytics service has stopped/not running.

2. Run the following commands (in Primary node) to check and remove the invalid "UserWrapper -> allowAllResources" for affected rows in postgres repl db kv_auth table:

   • Show before/after for rows containing invalid UserWrapper -> allowAllResources json element where objecttype = 'User-'
     su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c \"SELECT objecttype,jsonb_pretty(col__kv_strvalue::jsonb) AS jsonbefore, jsonb_pretty((col__kv_strvalue::jsonb) #- '{UserWrapper,allowAllResources}') AS jsonafter FROM kv_auth WHERE objecttype ='User-' AND (col__kv_strvalue::jsonb)->'UserWrapper' ? 'allowAllResources';\""
      
   • Delete invalid UserWrapper -> allowAllResources json element from all rows where objecttype = 'User-'
     su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c \"UPDATE kv_auth SET col__kv_strvalue = (col__kv_strvalue::jsonb) #- '{UserWrapper,allowAllResources}' WHERE objecttype ='User-' AND (col__kv_strvalue::jsonb)->'UserWrapper' ? 'allowAllResources';\""

3. After fixing the affected kv_auth UserWrapper rows, manually start analytics service on primary node (systemctl start analytics) and the upgrade should continue and complete successfully thereafter.

Additional Information

Required to execute a psql command during the upgrade to fix the UserWrapper rows having an invalid/obsolete allowAllResources immediate child element.

Impact/Risks:

Aria Operations upgrade gets stuck with analytics service failing to start.