"Search service is currently unavailable, please restart using NSX CLI command" and "Failed to get report - An unknown error has occurred" on NSX UI after upgrade

Products

• VMware NSX

Issue/Introduction

• NSX UI is reporting an error:
  
  Computer Managers list failed: Search service is currently unavailable, please restart using NSX CLI command 'restart service search'
  
  

• NSX UI is reporting an error:
  
  Failed to get report - An unknown error has occurred
Failed to get "Certificate" report - An unknown error has occurred
  
  
  
  
• On the NSX manager, /var/log/syslog has logs similar to this example 
   
  <Date>T02:44:18.701Z Manager NSX 5699 - [nsx@6876 comp="nsx-manager" errorCode="PM513002" level="ERROR" subcomp="manager"] Cannot fetch inventory details: Search service is currently unavailable, please restart using NSX CLI command 'restart service search'.
<Date>T02:35:21.471Z Manager systemd 1 - -  Failed to start search: VMware NSX Search.
<Date>T03:10:09.579Z Manager NSX 4753 - [nsx@6876 comp="nsx-manager" errorCode="MP60524" level="ERROR" subcomp="idps-reporting"] [Search: WatchDog] Could not connect to OpenSearch
  
  
• On the NSX Manager, /var/log/search/opensearch.log has logs similar to this example 
  
  org.opensearch.OpenSearchException: failed to bind service
 at org.opensearch.node.Node.<init>(Node.java:965) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) [opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) [opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) [opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-1.2.3.jar:1.2.3]
 at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) [opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) [opensearch-1.2.3.jar:1.2.3]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_372]
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_372]
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
 at com.vmware.nsx.service.Main.main(Main.java:23) [nsx-opensearch-1.0.jar:?]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_372]
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_372]
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
 at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349) [wrapper.jar:3.5.41]
 at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
Caused by: java.nio.file.AccessDeniedException: /nonconfig/search/nodes/0
 


Environment

VMware NSX 4.2.3

Cause

This issue can occur when the search symlink in /nonconfig/ points incorrectly to /config/search

Resolution

This is a known issue impacting VMware NSX.

 

The expected behavior is the symlink /nonconfig/search should point to nsx-search.

 

To resolve this, log into each NSX manager and elevate to root access:  st en

 

1. Stop below services on manager node: 

/etc/init.d/proton stop

/etc/init.d/cm-inventory stop

/etc/init.d/phonehome-coordinator stop

/etc/init.d/idps-reporting-service stop

/etc/init.d/search stop

2. Remove symlink "/nonconfig/search" 

       

unlink /nonconfig/search

 

3. Start search service first.

 

         /etc/init.d/search start

 

4. Start the remaining services         

/etc/init.d/proton start

/etc/init.d/cm-inventory start

/etc/init.d/phonehome-coordinator start

/etc/init.d/idps-reporting-service start

 

5. Resync NSX Managers

 

su admin -c start search resync all 

 

Once the above procedure is followed, nsx-search would point to as shown below