Error: Exception while searching during pipeline execution com.vmware.nsx.management.search.common.exceptions.SearchException: Invalid Query (Error code: 60576)

Products

VMware NSX

Issue/Introduction

When navigating the NSX UI, Networking -> Tier-0 Gateways/Tier-1 Gateways page shows.....

"Error: Exception while searching during pipeline execution com.vmware.nsx.management.search.common.exceptions.SearchException: Invalid Query (Error code: 60576)"

The following log patterns will also match:

/var/log/proton/policy-ui.log

- [nsx@6876 comp="nsx-manager" level="INFO" reqId="######-####-####-####-############" subcomp="manager" username="user@domain"] {"user":"user@domain","message":"Api Errors->","messageData":{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"Bad Request","url":"https://nsx-vip/policy/api/v1/search/aggregate?page_size=50&cursor=0&sort_by=display_name&sort_ascending=true","ok":false,"name":"HttpErrorResponse","message":"Http failure response for https://nsx-vip/policy/api/v1/search/aggregate?page_size=50&cursor=0&sort_by=display_name&sort_ascending=true: 400 Bad Request","error":{"details":"com.vmware.nsx.management.search.common.exceptions.SearchException: Invalid Query","httpStatus":"BAD_REQUEST","error_code":60576,"module_name":"nsx-search","error_message":"Error: Exception while searching during pipeline execution com.vmware.nsx.management.search.common.exceptions.SearchException: Invalid Query (Error code: 60576)","error_data":{"status":400}}},"level":"Error","browser":"[...]","time":"Tue Jan 01 2025 12:00:00 UTC+0000","location":"/app/networks/networks/home/networks"}

/var/log/proton/nsxapi.log

2025-01-01T12:00:00.853Z  INFO http-nio-127.0.0.1-7440-exec-32 NsxBaseRestController 5204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/search/aggregate?page_size=50&cursor=0&sort_by=display_name&sort_ascending=true caused by exception com.vmware.nsx.management.search.querypipeline.exceptions.QueryPipelineException:  {"moduleName":"nsx-search","errorCode":60576,"errorMessage":"Exception while searching during pipeline execution","details":"com.vmware.nsx.management.search.common.exceptions.SearchException: Invalid Query"}

/var/log/search/opensearch.log

[2025-01-01T12:00:00,846][DEBUG][o.o.a.s.TransportSearchAction] [nsx_node] [gEfzmkoxRnq-k14NJeTxxQ][nsx_policy_policyedgecluster][0]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[nsx_policy_policyedgecluster], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=true, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":50000,"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"query":"resource_type:(PolicyEdgeCluster) AND path:(\"/infra/sites/default/enforcement-points/default/edge-clusters/######-####-####-####-############\")","default_field":"_custom_all","fields":[],"type":"best_fields","default_operator":"and","analyzer":"default","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"lenient":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"match_all":{"boost":1.0}}],"should":[{"bool":{"must":[{"query_string":{"query":"path.aggregation:\\/*","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"bool":{"must":[{"query_string":{"query":"(!resource_type:(Alarm OR MonitoringEvent OR IDSEvent OR ConsolidatedRealizedStatus OR VirtualMachine OR VirtualNetworkInterface OR GenericPolicyRealizedResource) AND !(_exists_:path))","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"must_not":[{"match":{"_meta.isOverridden":{"query":true,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["path","display_name","parent_path"],"excludes":["nested_tags","_shared_meta","_meta.span_path"]},"sort":[{"_doc":{"order":"asc"}}],"track_total_hits":2147483647}, cancelAfterTimeInterval=null}] lastShard [true]
org.opensearch.transport.RemoteTransportException: [nsx_node][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.
        at org.opensearch.search.DefaultSearchContext.preProcess(DefaultSearchContext.java:244) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.search.query.QueryPhase.preProcess(QueryPhase.java:135) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.search.SearchService.createContext(SearchService.java:836) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:501) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.search.SearchService.access$500(SearchService.java:154) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:474) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:71) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:86) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:57) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_382]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_382]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_382]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

A similar message may make different parts of the UI actions unavailable.

Environment

VMware NSX

Cause

There could be different underlying causes for this particular error. One cause is, that the indices in the search database are not having the correct settings applied, out-of-sync or corrupt (for example, during unexpected reboots).

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

If you are contacting Broadcom support about this issue, in order to aid a timely response and resolution, please provide the following:

NSX Manager log bundles.
Text of any error messages seen in the NSX GUI and the screenshot.

Handling Log Bundles for offline review with Broadcom support