"Failed to fetch inventory details" Error on NSX UI "Search service health status is degraded" in VMware NSX 4.2

Products

• VMware NSX

Issue/Introduction

• NSX web user interface may be inaccessible.
• NCP and other automated workflows may fail.
• In the NSX Manager you may see similar entries in the /var/log/search/opensearch.log

[2024-10-26T17:01:31,974][WARN ][o.o.i.e.Engine] [nsx_node] [nsx_manager_lbservice][0] failed engine [merge failed]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge
        at org.opensearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2631) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge

• In the NSX Manager you may see similar entries in the /var/log/syslog

2024-10-28T05:58:26.471Z ###.######.### NSX 5175 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'nsx-search' Operation:'GET@/api/v1/search' Operation status: 'failure' Error: Search service health status is degraded. Run the 'start search recover' CLI command on NSX node: <nsx-manager-node>(<nsx-manager-ip>).

• On the NSX Web GUI, there is an error stating "Failed to fetch inventory details. Please contact the administrator. Error: Search service health status is degraded. Run the 'start search recover' CLI command on NSX node <nsx-manager FQDN/IP>. (Error code:513002)"

Note :To be able to access the log files of the NSX manager as root user, if ssh as root is not permitted, follow the steps below:

• Login to the NSX manager as user admin
• Switch to the root user account by running the command st en

Environment

VMware NSX 4.2.x

Cause

Search indices get corrupted which can happen due to storage/hardware issues.

Resolution

This issue is resolved in VMware NSX 4.2.2 and VCF NSX 9.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround: 

On each NSX Manager node, as the admin user, run the command 'start search recover' to repair the indices. 

In case of storage issues on the current datastore where the NSX manager exists, if the above step does not help then try performing Storage vMotion of the NSX manager to another datastore and then try  'start search recover' 

Additional Information

To know more about troubleshooting storage issues, refer to the KB :- https://knowledge.broadcom.com/external/article/367437 

To know more about  troubleshooting hardware faulty, refer to the KB :- https://knowledge.broadcom.com/external/article/336323