NSX UI Certificates page fails to load and "Index out of sync" error

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T 3.2.x and NSX 4.x
• The environment has been upgraded in the past from a version prior to 3.x.
• NSX UI Certificate page will not load
• NSX UI search is not working
• NSX UI reports an error

  Error: Index out of sync, please resync via 'start search resync {0}'. (Error code: 60516)

• In some cases UI may also report the warning

  System is still in uncertain state please reload

• Logs will have an exception in /var/log/proton/nsxapi.log similar to this example

2023-03-03T12:45:32.074Z ERROR pool-776-thread-1 UfoIndexingServiceImpl 13446 - [nsx@6876 comp="nsx-manager" errorCode="MP60503" level="ERROR" subcomp="manager"] [Indexing: BatchProcessing] The Bulk indexing request could not be processed: org.elasticsearch.action.bulk.BulkRequest@7563b312
org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: index is missing;2: index is missing;3: index is missing; 

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
VMware NSX 4.0.0.1

Cause

NSX environments that started on older versions of NSX, typically pre-3.x may still have old certificates in the system.
NSX 3.2.x and 4.x fails to handle certificates with an old service type as they are not used on these versions.

Resolution

This is a known issue, currently there is no resolution.

Workaround:
Identify if the system has any certificates using old service types by running the API to list certs

GET /api/v1/trust-management/certificates 

Check for the "service_types" field.

Certificates with any of the following service_types will cause this issue
 

"Node API Certificate"
"MGMT_PLANE_DNE"
"Client Authentication"
"Cluster API Certificate" 

If the certificate is in use, make note of the node UUID in the "used_by" field.

The certificate will first need to be released from the node and then deleted

ssh to any NSX Manager as root user and for each cert 

curl -k -X POST -H "Content-Type: application/json" -H 'X-NSX-Username:admin' -H 'X-NSX-Groups:superuser' -d '{"node_id":"<NODE_UUID>"}'  "http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/<CERT_UUID>?action=release"
curl -k -X DELETE -H "Content-Type: application/json" -H 'X-NSX-Username:admin' -H 'X-NSX-Groups:superuser'  "http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/CERT_UUID"

Note if the certificates are used by a Principle Identity solution such as Tanzu, it may be necessary to delete the associated cluster deployment and redeploy it to free up the problem certificate.