NSX Manager is reporting search sync issues

Products

VMware NSX

Issue/Introduction

NSX Manager is reporting search service is unavailable
Issue is reported by one of the NSX manager node
Reboot of the NSX manager node did not resolve the issue
Issue is reported in NSX manager version 3.2.2
Error look similar to the below screenshot
Error: Failed to fetch Networking details. Please contact the administrator. Error: Search service is currently unavailable, please restart using 'restart service search' and resync using 'start search resync all' CLI commands on the NSX appliance. (Error code: 513002)

Log validation on GM.

Log full path: /var/log/syslog

<NSX-GM># cat /var/log/syslog | grep -i "Cannot fetch inventory details\|start search"
2025-07-28T10:49:43.446Z <NSX GM FQDN> NSX 4203 - [nsx@6876 comp="global-manager" errorCode="PM513002" level="ERROR" subcomp="global-manager"] Cannot fetch inventory details: Re-creating indexes, this may take some time.
2025-07-28T10:50:57.606Z  <NSX GM FQDN> NSX 4203 - [nsx@6876 comp="global-manager" errorCode="PM513002" level="ERROR" subcomp="global-manager"] Cannot fetch inventory details: Re-creating indexes, this may take some time.
2025-07-28T10:51:24.271Z <NSX GM FQDN> NSX 4203 - [nsx@6876 comp="global-manager" errorCode="PM513002" level="ERROR" subcomp="global-manager"] Cannot fetch inventory details: Re-creating indexes, this may take some time.

Other possible symptom - Metadata index issue.(Refer resolution 2 for the below issue).

Due to incorrect template applied for metadata index, these indices go into yellow state and hence destabilized "elasticsearch".

Below log reports on one of the LM:

cat var/log/search/elasticsearch.log
2025-07-28T19:37:56,147][INFO ][o.e.m.j.JvmGcMonitorService] [nsx_node] [gc][39] overhead, spent [295ms] collecting in the last [1s]
[2025-07-28T19:39:57,306][WARN ][r.suppressed             ] [nsx_node] path: /monitoring_metadata/_doc/monitoring, params: {index=monitoring_metadata, id=monitoring}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [monitoring_metadata][_doc][monitoring]: routing [null]]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:210) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:103) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:62) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:491) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.rest.action.document.RestGetAction.lambda$prepareRequest$0(RestGetAction.java:97) ~[elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:113) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:337) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:317) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:367) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:296) [elasticsearch-7.1.1.jar:7.1.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-7.1.1.jar:7.1.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.1.1.jar:7.1.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.1.1.jar:7.1.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]

Environment

VMware NSX-T 3.2.2

Cause

This issue occurs due to the search service is not running

Resolution

Resolution 1:

To resolve the issue, run 'restart service search' and resync using 'start search resync all' CLI commands on the problematic NSX Manager appliance.
Note: NSX Manager may return an error message "% An error occurred while re-syncing search service" after issuing the command 'start search resync all'. Please allow a few minutes for the search service to fully restart and stabilize before initializing the resync operation.

Resolution 2 for "Metadata index issue reported on one of the LM":

Perform a restart proton service using cli command "service proton restart" by login as root account on problematic LM where the Metadata index issue reported.
Then, Run the following command on LM "start search resync all". Once the command line execution completed, perform the same command "start search resync all" on other 2 LM's.
Note: NSX Manager may return an error message "% An error occurred while re-syncing search service" after issuing the command 'start search resync all'. Please allow a few minutes for the search service to fully restart and stabilize before initializing the resync operation.

Additional Information

If the above steps do not resolve the issue, raise a support ticket with Broadcom support selecting NSX as the product.

Handling Log Bundles for offline review with Broadcom support.