NSX Manager is reporting search sync issues after upgrade to 4.2.3
search cancel

NSX Manager is reporting search sync issues after upgrade to 4.2.3

book

Article ID: 411474

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX Manager is reporting search service is unavailable

  • Issue observed on all three manager nodes.

  • Rolling reboot of the NSX managers did not fix the issue

  • Error look similar to the below snippet

    Computer Managers list failed: Search service is currently unavailable, please restart using NSX CLI command 'restart service search'

  • Log validation of NSX manager 
     
  • root@mgr1:/var/log# less syslog | grep -i "Cannot fetch inventory" 2025-09-24T02:44:18.701Z mgr1.xxxx.com NSX 5699 - [nsx@6876 comp="nsx-manager" errorCode="PM513002" level="ERROR" subcomp="manager"] Cannot fetch inventory details: Search service is currently unavailable, please restart using NSX CLI command 'restart service search'.
  • root@mgr1:/var/log# less syslog | grep -i "start search"
2025-09-24T02:35:21.471Z mgr1.xxxx.com systemd 1 - -  Failed to start search: VMware NSX Search.
  • root@mgr1:/var/log# tail -f syslog

    2025-09-24T03:10:09.579Z nsxtmgr1.xxx.xxx NSX 4753 - [nsx@6876 comp="nsx-manager" errorCode="MP60524" level="ERROR" subcomp="idps-reporting"] [Search: WatchDog] Could not connect to OpenSearch
  • /var/log/search/opensearch.log

    org.opensearch.OpenSearchException: failed to bind service
     at org.opensearch.node.Node.<init>(Node.java:965) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) [opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) [opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) [opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-1.2.3.jar:1.2.3]
     at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) [opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) [opensearch-1.2.3.jar:1.2.3]
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_372]
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_372]
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
     at com.vmware.nsx.service.Main.main(Main.java:23) [nsx-opensearch-1.0.jar:?]
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_372]
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_372]
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
     at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349) [wrapper.jar:3.5.41]
     at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
    Caused by: java.nio.file.AccessDeniedException: /nonconfig/search/nodes/0
     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_372]
     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_372]
     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_372]
     at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_372]
     at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_372]
     at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_372]
     at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_372]
     at org.opensearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:301) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:239) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:299) ~[opensearch-1.2.3.jar:1.2.3]
     at org.opensearch.node.Node.<init>(Node.java:414) ~[opensearch-1.2.3.jar:1.2.3]

Environment

VMware NSX 4.2.3

Cause

symlink  /nonconfig/search was pointing to /config/search

Resolution

The expected behavior is the symlink /nonconfig/search should point to nsx-search
 
Log into each NSX manager and elevate to root access:  st en
 
1. Stop below services on manager node: 

/etc/init.d/proton stop

/etc/init.d/cm-inventory stop

/etc/init.d/phonehome-coordinator stop

/etc/init.d/idps-reporting-service stop

/etc/init.d/search stop

2. Remove symlink "/nonconfig/search" 
       
unlink /nonconfig/search
 
3. Start search service first.
 
         /etc/init.d/search start
 
4. Start the remaining services         

/etc/init.d/proton start

/etc/init.d/cm-inventory start

/etc/init.d/phonehome-coordinator start

/etc/init.d/idps-reporting-service start

 
5. Resync NSX Managers
 
su admin -c start search resync all 
 
Once the above procedure is followed, nsx-search would point to as shown below 

Powered by Wolken Software