NSX Manager Search service status is not UP
search cancel

NSX Manager Search service status is not UP

book

Article ID: 385577

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

  • NSX Manager Search service status is Degraded/Down
  • The NSX Search service status shows as Degraded or Down in the NSX UI. This condition impacts the ability to search for inventory objects and may result in outdated or missing data in the Manager dashboard.
  • Any of the below error messages may appear:
    • Error messages:
      • Index is currently out of sync for ''resource_type:<list of resource types>'', system is trying to recover
      • Search index is out of sync for ''resource_type:<list of resource types>'' Run the ''start search resync {1}'' CLI command on the NSX appliance to resync. If the issue persists, contact Broadcom Support
      • Search framework initialization failed, restart the service via ''restart service <service name>’’.
      • Search service is currently unavailable, restart using ''restart service search'' and resync using ''start search resync all'' CLI commands on the NSX appliance.
      • Search service is currently unavailable, restart using NSX CLI command ''restart service search’’.
      • Search service health status is degraded. Run the ''start search recover'' CLI command on NSX node: <node ip>.”
      • The service is not responding.

The solution will depend on the specific error message received.

Environment

  • VMware NSX
  • VMware Cloud Foundation (VCF) 9.x
  • VCF Networking

Cause

  1. Degraded status:
    The search service is degraded if there are failures in indexing one or more entities.
    The search service enters a degraded state when indexing failures occur for one or more entities.
    If indexing retries (up to ten times) fail, the service enters a non-recoverable state.

  2. Down status:
    The search service can be down due to one of the following causes:

    a. Search framework initialization failed.
    b. Search service is unavailable and resync is required.
    c. The search service is unavailable.
    d. The search indices are in an unhealthy state.
    e. The manager service is down

 

Resolution

Steps to Resolve

For VCF 9.0 and higher

Recommended Actions:

  1. Degraded status: The search service is degraded if there are failures in indexing one or more entities. In such cases, one of the following two error messages may appear:
    1. Indexing failures in recovery state
          1. Error message: Index is currently out of sync for ''resource_type:<list of resource types>'', system is trying to recover.
          2. Recommendation: This indicates an error in indexing one or more entity types. The failures in indexing are retried ten times. In case it's an intermittent failure this message should go away in the next couple of retries and the search service would get into a healthy state. In case the retries do not succeed it gets into the non-recoverable failure state.
    2. Non-recoverable indexing failures
          1. Error message: Search index is out of sync for ''resource_type:<list of resource types>'' Run the ''start search resync {1}'' CLI command on the NSX appliance to resync. If the issue persists, contact Broadcom Support.
          2. Recommendation: This indicates a non-recoverable error in indexing one or more entity types. Try to resync by executing the CLI and in case the issue continues to persist reach out to the support for help on this issue.


  2. Down status: The search service can be down due to one of the following causes:
    1. Search framework initialization failed
            1. Error message: Search framework initialization failed, restart the service via ''restart service <service name>’’.

            2. Recommendation: This error indicates a failure in the search framework initialization task. As advised in the error message to restart the service.

      1. Search service is unavailable and resync is required  

            1. Error message:  Search service is currently unavailable, restart using ''restart service search'' and resync using ''start search resync all'' CLI commands on the NSX appliance.

            2. Recommendation: This indicates an operational issue with the search service. As indicated in the message restart the search service by executing the CLI command ''restart service search'' and then resync the data by executing ''start search resync all''

      2. The search service is unavailable
            1. Error message: Search service is currently unavailable, restart using NSX CLI command ''restart service search’’.
            2. Recommendation: This indicates that the search service is not running and must be restarted.
      3. The search indices are in an unhealthy state

            1. Error message: Search service health status is degraded. Run the ''start search recover'' CLI command on NSX node: <node ip>.”

            2. Recommendation: As indicated in the error message to execute the CLI to recover the search service.

      4. The manager service is down
            1. Error message: The service is not responding.
            2. Recommendation: This indicates that the manager service is down. When the manager is down the search service is marked as down too. Restart the manager service using the CLI "restart service manager".
      5. Restart of services fail
            1. Rolling Reboot of manager nodes

Additional Information

Impact:

In case the search service is degraded/down it impacts the functioning of the NSX UI