Search service health status is degraded in VMware NSX 4.2

Products

VMware NSX

Issue/Introduction

NSX web user interface may be inaccessible.
NCP and other automated workflows may fail.
In the NSX Manager you may see similar entries in the /var/log/search/opensearch.log

[2024-10-26T17:01:31,974][WARN ][o.o.i.e.Engine] [nsx_node] [nsx_manager_lbservice][0] failed engine [merge failed]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge
        at org.opensearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2631) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge

In the NSX Manager you may see similar entries in the /var/log/syslog

2024-10-28T05:58:26.471Z ###.######.### NSX 5175 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'nsx-search' Operation:'GET@/api/v1/search' Operation status: 'failure' Error: Search service health status is degraded. Run the 'start search recover' CLI command on NSX node: <nsx-manager-node>(<nsx-manager-ip>).

On the NSX Web GUI, there is an error stating "Failed to fetch inventory details. Please contact the administrator. Error: Search service health status is degraded. Run the 'start search recover' CLI command on NSX node:"

Note :- To be able to access the log files of the NSX manager, follow the steps below:-

Login to the NSX manager as user admin
Switch to the root user account by running the command st en

Environment

VMware NSX 4.2.x

Cause

Search indices get corrupted which can happen due to storage/hardware issues.

Resolution

This is a known issue impacting VMware NSX 4.2

Workaround:

On each NSX Manager node, as the admin user, run the command 'start search recover' to repair the indices.

In case of storage issues on the current datastore where the NSX manager exists, if the above step does not help then try performing Storage vMotion of the NSX manager to another datastore and then try 'start search recover'

Additional Information

To know more about troubleshooting storage issues, refer to the KB :- https://knowledge.broadcom.com/external/article/367437

To know more about troubleshooting hardware faulty, refer to the KB :- https://knowledge.broadcom.com/external/article/336323