NSX Search APIs are continuously failing after upgrade, users are not able to deploy any new pods

Products

• VMware NSX

Issue/Introduction

• Search re-indexing is taking a long time on setups with a large number of Logical Ports. The Search API gives the error during the period of Indexing.
• After upgrade or restart, the Search API may return 400 error with Error code 60508, "Re-creating indexes, this may take some time."
• User will see the error on accessing the Search API and UI till reindexing is complete. 

The issue only occurs during reindexing and only for large scale of certain entities like Logical Ports, and will recover as the reindexing is complete.

Depending on the scale of the system, the Search API and the UI are unusable until the re-indexing is complete.

Log example from /var/log/search/search-manager.* on Manager nodes:
2020-12-02T07:12:53.402Z  INFO task-executor-7 IndexingServiceImpl 12963 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 0 LogicalDhcpServer for indexing took 1 ms
2020-12-02T07:25:49.745Z  INFO task-executor-7 IndexingServiceImpl 12963 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 70865 LogicalPort for indexing took 776233 ms

Cause

The error is seen due to re-indexing going on as part of the upgrade process. The indexing time can vary on the number of entities and system load.

Resolution

This issue is resolved in NSX-T 3.2.2.

Additional Information

There is a similar issue where Search APIs can return errors after an upgrade to 3.2.1 due to Elastic Search receiving requests before it is fully recovered after a restart. In this case, the system will be auto-recovered once re-indexing is complete.
This issue is resolved in NSX-T 3.2.2.

Impact/Risks:
Users will have to wait for the re-indexing to be completed. The time will vary based on the scale and load on the system.