Data Access Issue on NSX UI Post Upgrade: Error "Search service is currently unavailable, please restart using 'restart service search.'"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Affected Versions: This issue is known to occur in versions earlier than 3.2.2.
• Upgrading from a version earlier than 3.2.2 may result in this issue if there are difficulties encountered during the deletion of indices.
• In the /var/log/search/search-inventory.log user would observe following error

  2023-xx-xxT13:11:59.390Z WARN pool-46-thread-1 UfoIndexingServiceImpl 70113 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] [Indexing] Error while deleting indices
  2023-xx-xxT13:11:59.390Z WARN pool-46-thread-1 UfoIndexingServiceImpl 70113 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] [Indexing] Search service is not available due errors while deleting indices

•  Relevant log location in NSX Manager to catch the error message. 
  /var/log/search/search-manager.log
  /var/log/search/elasticsearch.log

Environment

VMware NSX-T

Cause

• The issue is caused by a faulty setting on a few metadata indices in versions earlier than 3.2.2. This faulty setting can lead to failures in updates or deletes on these indices.
• As part of the upgrade process, a complete reindexing is performed. During this reindexing, all indices from the older version are first deleted.
• In the event of unsuccessful deletion of indices during the upgrade process from a version prior to 3.2.2, specifically due to the presence of faulty indices, the search service will be flagged as unavailable.
• As a direct result of the search service being marked as unavailable, users attempting to access data on the UI will encounter the error message "Search service is currently unavailable. Please restart by using the 'restart service search' command."

Resolution

This issue has been addressed and fixed in NSX versions 3.2.2 and 4.0.1.

Workaround:

1. Stop the proton service using the following command 
   nsxmgr# service proton stop

2. Restart the search service using the following command
   nsxmgr# service search restart
   nsxmgr# service search status 

3. Start the proton service using the following command 
   nsxmgr# service proton start
   nsxmgr# service proton status

Additional Information

Impact/Risks:
When the search service is unavailable, users may encounter a persistent error message on the UI, preventing them from viewing data. The error message will state "Search service is currently unavailable. Please restart by using the 'restart service search' command