Searching fails in the NSX Manager UI stating that the Search index is out of sync.

Products

VMware NSX

Issue/Introduction

Following an upgrade from NSX 3.x to NSX 4.x, the NSX UI's Alarm page is no longer loading.
The following message is displayed when attempting to reach the NSX Alarm page:
"Search index is out of sync. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support.","details":"INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [all]"
In the VMware NSX Proton logs (/var/log/proton/nsxapi.log) the following errors can be seen:

"2024-08-22T22:41:03.997Z WARN pool-736-thread-1 CapacityDataCollectionServiceImpl 274408 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Encountered Exception while invoking getCapacityData for GROUPING
com.vmware.nsx.management.search.common.exceptions.SearchException: Search index is out of sync. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support.
com.vmware.nsx.management.search.consistency.SearchSyncManagerImpl.validateConsistency(SearchSyncManagerImpl.java:128)
com.vmware.nsx.management.search.consistency.SearchSyncManagerImpl.validateConsistencyForResourceTypes(SearchSyncManagerImpl.java:86)
com.vmware.nsx.management.search.consistency.SearchSyncManagerImpl.validateConsistencyForQuery(SearchSyncManagerImpl.java:67)
com.vmware.nsx.management.search.facade.SearchFacadeImpl.search_aroundBody4(SearchFacadeImpl.java:99)
com.vmware.nsx.management.search.facade.SearchFacadeImpl$AjcClosure5.run(SearchFacadeImpl.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:168)
com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:422)
com.vmware.nsx.management.common.api.FacadeInterceptorAspect.annotatedMethod(FacadeInterceptorAspect.java:80)
com.vmware.nsx.management.search.facade.SearchFacadeImpl.search(SearchFacadeImpl.java:95)
com.vmware.nsx.management.groupingobjects.ufo.telemetry.GroupingTelemetryProviderImpl.executeSearchRequest(GroupingTelemetryProviderImpl.java:258)
com.vmware.nsx.management.groupingobjects.ufo.telemetry.GroupingTelemetryProviderImpl.getCapacityConfigUsingSearch(GroupingTelemetryProviderImpl.java:213)
com.vmware.nsx.management.groupingobjects.ufo.telemetry.GroupingTelemetryProviderImpl.getCapacityUsage(GroupingTelemetryProviderImpl.java:199)
sun.reflect.GeneratedMethodAccessor4748.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.vmware.nsx.management.phonehome.service.impl.CapacityDataCollectionServiceImpl$1.call(CapacityDataCollectionServiceImpl.java:248)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)"

"2024-08-22T22:41:02.540Z ERROR http-nio-127.0.0.1-7440-exec-8 NsxBaseRestController 274408 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] 0
java.lang.ArrayIndexOutOfBoundsException: 0
        at com.vmware.nsx.management.aggservice.appliancestats.service.InterfaceStatsServiceImpl.getInterfaceStats(InterfaceStatsServiceImpl.java:378) ~[?:?]
        at com.vmware.nsx.management.aggservice.appliancestats.service.InterfaceStatsServiceImpl.getFabricNodeInterfaceStats(InterfaceStatsServiceImpl.java:235) ~[?:?]
        at com.vmware.nsx.management.aggservice.appliancestats.facade.ApplianceStatsFacadeImpl.readTransportNodeInterfaceStatistics_aroundBody10(ApplianceStatsFacadeImpl.java:95) ~[?:?]
        at com.vmware.nsx.management.aggservice.appliancestats.facade.ApplianceStatsFacadeImpl$AjcClosure11.run(ApplianceStatsFacadeImpl.java:1) ~[?:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[?:?]
        at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:168) ~[?:?]"

"2024-08-21T20:18:29.813Z ERROR UfoIndexer-BatchExecutor-search_manager-0 UfoGenericConverter 296766 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DataProvider] Could not convert UFO object to Dto UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='InternalGroup', streamTag=MANAGER}, identifier=uuid {
  left: ####
  right: ####
}
}
java.lang.reflect.InvocationTargetException: null
        at sun.reflect.GeneratedMethodAccessor1785.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
        at com.vmware.nsx.management.search.provider.UfoGenericConverter.internalConvert(UfoGenericConverter.java:102) ~[?:?]
        at com.vmware.nsx.management.search.provider.UfoGenericConverter.convertToDataToIndex(UfoGenericConverter.java:71) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processForIndexing(UfoIndexingServiceImpl.java:625) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processUfoObjectForIndexing(UfoIndexingServiceImpl.java:557) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.access$1(UfoIndexingServiceImpl.java:555) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl$1.process(UfoIndexingServiceImpl.java:526) ~[?:?]
        at com.vmware.nsx.management.search.configuration.DataStore.processDynamicTxBatch(DataStore.java:120) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processBatchForIndexing(UfoIndexingServiceImpl.java:514) ~[?:?]
        at com.vmware.nsx.management.search.manager.UfoIndexer.processChunk(UfoIndexer.java:220) ~[?:?]
        at com.vmware.nsx.management.search.manager.UfoIndexer.lambda$2(UfoIndexer.java:169) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
Caused by: java.lang.IllegalArgumentException: No enum constant com.vmware.nsx.management.policy.policyframework.model.GroupMemberType.INVALID"

In the VMware NSX search logs (/var/log/search/search-manager.log) the following errors can be seen:

"2024-09-01T11:47:58.813Z ERROR UfoIndexer-BatchExecutor-search_manager-3 UfoGenericConverter 86181 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DataProvider] Could not convert UFO object to Dto UfoObject{operationType=INDEX, descriptor=IndexingTypeDescriptor{tableName='InternalGroup', streamTag=MANAGER}, identifier=uuid {
  left: ####
  right: ####
}
, timestamp.sequence =5396803014, timestamp.epoch=291}
java.lang.reflect.InvocationTargetException: null
        at sun.reflect.GeneratedMethodAccessor4925.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
        at com.vmware.nsx.management.search.provider.UfoGenericConverter.internalConvert(UfoGenericConverter.java:102) ~[?:?]
        at com.vmware.nsx.management.search.provider.UfoGenericConverter.convertToDataToIndex(UfoGenericConverter.java:71) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processForIndexing(UfoIndexingServiceImpl.java:625) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processUfoObjectForIndexing(UfoIndexingServiceImpl.java:557) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.access$1(UfoIndexingServiceImpl.java:555) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl$1.process(UfoIndexingServiceImpl.java:526) ~[?:?]
        at com.vmware.nsx.management.search.configuration.DataStore.processDynamicTxBatch(DataStore.java:120) ~[?:?]
        at com.vmware.nsx.management.search.service.impl.UfoIndexingServiceImpl.processBatchForIndexing(UfoIndexingServiceImpl.java:514) ~[?:?]
        at com.vmware.nsx.management.search.manager.UfoIndexer.processChunk(UfoIndexer.java:220) ~[?:?]
        at com.vmware.nsx.management.search.manager.UfoIndexer.lambda$2(UfoIndexer.java:169) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
Caused by: java.lang.IllegalArgumentException: No enum constant com.vmware.nsx.management.policy.policyframework.model.GroupMemberType.INVALID"

Running the admin CLI command: "start search resync all" doesn't resolve the issue.

Environment

NSX 4.x
NSX-T Datacenter

Cause

There is inconsistent data between 'Management Plane' and 'Policy' Databases. The internal static membership inconsistency prevents services from running correctly, which causes parts of the NSX UI to not load properly and any searches in the UI to fail with an error message.

Resolution

This is a known issue impacting VMware NSX. To resolve the issue, the Group(s) with database issues will need to be identified, removed, and re-created.

Additional Information

This cannot be reproduced starting with VMware NSX 4.1 release as Manager APIs are deprecated, and Policy APIs are only used to update Group. The issue can still be present in environments that have been upgraded from below 4.1.

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager log bundles. *Ensure log date range covers the full date of the event(s) being investigated. When in doubt, retrieve logs for all time.
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support