Post upgrade from 3.2.2 to 4.2.3.1, alarms page is not loading on NSX UI.

Products

VMware NSX

Issue/Introduction

Search indexes of some objects are out of sync.
Customer is unable to use alarms UI 4.2.3.1

The following message is displayed when attempting to reach the NSX Alarm page:

Error: Search index is out of sync for 'IP Set, MACSet, NSGroup'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. (Error code: 60516)

In the VMware NSX Search Manager logs (/var/log/search/search-manager.log) the following errors can be seen, please note that the number of batch items (groups) that may have the problem.

YYYY-MM-DDTHH:MM:SS.613Z ERROR UfoIndexer-search_manager-0 UfoGenericConverter 2186402 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DataProvider] Could not convert UFO object to Dto UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='InternalGroup', streamTag=MANAGER}, identifier=uuid {
  left: ##############
  right: ##############
}
}
java.lang.reflect.InvocationTargetException: null
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_372]
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_372]
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
 at com.vmware.nsx.management.search.provider.UfoGenericConverter.internalConvert(UfoGenericConverter.java:97) ~[?:?]
 at com.vmware.nsx.management.search.provider.UfoGenericConverter.convertToDataToIndex(UfoGenericConverter.java:68) ~[?:?]
 at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processForIndexing(UfoIndexingServiceImpl.java:563) ~[?:?]
 at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processUfoObjectForIndexing(UfoIndexingServiceImpl.java:533) ~[?:?]
 at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.access$1(UfoIndexingServiceImpl.java:527) ~[?:?]
 at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl$1.process(UfoIndexingServiceImpl.java:501) ~[?:?]
 at com.vmware.nsx.management.search.configuration.DataStore.processDynamicTxBatch(DataStore.java:101) ~[?:?]
 at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processBatchForIndexing(UfoIndexingServiceImpl.java:489) ~[?:?]
 at com.vmware.nsx.management.search.manager.UfoIndexer.processQueue(UfoIndexer.java:128) ~[?:?]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_372]
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_372]
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_372]
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_372]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
 at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]

Caused by: java.lang.NullPointerException
 at com.vmware.nsx.management.groupingobjects.ufo.converter.NSGroupExpressionConverterUtils.getNSGroupExpression(NSGroupExpressionConverterUtils.java:90) ~[?:?]
 at com.vmware.nsx.management.groupingobjects.providers.ufo.converter.VMIntentExpressionConverter.getNSGroupExpression(VMIntentExpressionConverter.java:124) ~[?:?]
 at com.vmware.nsx.management.groupingobjects.ufo.converter.NSGroupIntentConverter.toNSGroupExpressions(NSGroupIntentConverter.java:368) ~[?:?]
 at com.vmware.nsx.management.groupingobjects.ufo.converter.NSGroupIntentConverter.toNSGroupModel(NSGroupIntentConverter.java:134) ~[?:?]
 at com.vmware.nsx.management.search.dataprovider.GroupingObjectDataProvider.indexNSGroup(GroupingObjectDataProvider.java:189) ~[?:?]
 at com.vmware.nsx.management.search.dataprovider.GroupingObjectDataProvider.getNSGroupToIndex(GroupingObjectDataProvider.java:180) ~[?:?]
 ... 20 more

This error is because of the NullPointerException on some groups.
To identify the problematic group(s) use the below grep command to look for InternalGroup in /var/log/search/search-manager.log
```
grep -A 2 "identifier=uuid {" search-manager.log | grep -A 2 InternalGroup
```

Environment

NSX 4.2.x or lower.

Cause

The groups in customer's environment have condition expressions without any operator value.
The operator field is optional and groups can be created from API which looks to be the expected behaviour. However, search indexing is failing with NullPointerException during conversion which looks to be a known issue in older releases .
This issue is no longer applicable for >=9.0 as indexing of "InternalGroup" table is removed as part of MP API removal feature.

Resolution

This is a known issue in 4.2.3.

As a workaround, please follow below steps for identified groups.

Navigate to NSX UI
Move to Inventory Tab -> Groups
Search for the required group.
Edit the group.
Click on criteria and choose "Apply" button at the bottom.
Save the group.
Verify from the group GET API [GET /policy/api/v1/infra/domains/{domain-id}/groups/{group-id}] response that operator value is populated to equals.
Repeat steps 3-7 for both the aforementioned groups.
Run "start search resync all" from CLI on each of the manager nodes one after another.