NSX UI unable to load and displays "Unable to fetch details"

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• NSX UI unable to load and displays "Unable to fetch details"
• The following logs can be confirmed in NSX Manager:
  /var/log/elasticsearch.log
  
  [2022-12-03T18:17:21,159][WARN ][r.suppressed ] [nsx_node] path: /manager_metadata/_doc/manager, params: {index=manager_metadata, id=manager}
  org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [manager_metadata][_doc][manager]: routing [null]]
          at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.onFailure(TransportSingleShardAction.java:112) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) [elasticsearch-7.1.1.jar:7.1.1]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-7.1.1.jar:7.1.1]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
  Caused by: org.elasticsearch.transport.RemoteTransportException: [nsx_node][127.0.0.1:9300][indices:data/read/get[s]]
  Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]

Environment

VMware NSX-T Data Center 3.0

Cause

During restart of an appliance "search" service didn't have enough CPU resources to finish initialization within expected time threshold which caused "*_metadata" indices to fail to store latest indexing position. It caused further out of sync issue after services completed initialization.

Resolution

This issue has been fixed in NSX 4.2. The following workarounds are available:

Execute the following recovery instructions on each NSX node.

1. Stop all services which use "search" service
service phonehome-coordinator stop
service idps-reporting-service stop
service proton stop
service nsx-policy-manager stop

2. Remove stale *_metadata indices (it's runtime data and safe to delete)
curl -XDELETE 'http://localhost:9200/manager_metadata/'
curl -XDELETE 'http://localhost:9200/policy_metadata/'
curl -XDELETE 'http://localhost:9200/security_data_service_metadata/'
curl -XDELETE 'http://localhost:9200/monitoring_metadata/'

3. Restart "search" service to make sure it is fully initialized
service search restart
grep " started" /var/log/search/elasticsearch.log

4. Start phonehome-coordinator service
service phonehome-coordinator start
grep "Complete Indexing is finished" /var/log/phonehome-coordinator/phonehome-coordinator.log

5. Start idps service
service idps-reporting-service start
grep "Complete Indexing is finished" /var/log/idps-reporting/idps.log

6. Start proton service
service proton start
grep "Complete Indexing done" /var/log/search/search-manager.log

7. Start policy service
service nsx-policy-manager start
grep "Complete Indexing done" /var/log/search/search-policy.log

8. Verify UI is fully functional