Error code: 60516 - NSX Policy search API is failing - "Error: Search index is out of sync."

Products

• VMware NSX

Issue/Introduction

• A Policy API such as /policy/api/v1/search?query=resource_type:EdgeCluster%20AND%20display_name:edge-cluster-capacity is failing with the following error:

  "Error: Search index is out of sync. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [all] (Error code: 60516)"

• The response from the API reports a http 400 'bad request status' code.
• Also, this issue can be seen when using the NSX Manager UI.
• An equivalent Manager API such as /api/v1/search?query=resource_type:EdgeCluster%20AND%20display_name:edge-cluster-capacity succeeds.
• Log lines similar to the below are encountered on the NSX Manager in /var/log/search/search-manager.log

  ERROR UfoIndexer-BatchExecutor-search_manager-1 UfoGenericConverter 4792 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='ServiceConfig', streamTag=MANAGER}, identifier=uuid {
    left: 11111111111111111111
    right: 22222222222222222222
  }
  }
  Caused by: com.vmware.nsx.management.common.exceptions.ObjectNotFoundException
          at com.vmware.nsx.management.container.dao.IdentifiableProxyObjectDao.findByIdentifier(IdentifiableProxyObjectDao.java:400) ~[?:?]
          at com.vmware.nsx.management.edge.lrouter.service.LogicalRouterServiceImpl.getLogicalRouter(LogicalRouterServiceImpl.java:391) ~[?:?]
          at com.vmware.nsx.management.serviceconfig.converter.ServiceConfigConverter.toDtoWithTx(ServiceConfigConverter.java:219) ~[?:?]
          at com.vmware.nsx.management.serviceconfig.converter.ServiceConfigConverter.toDtoWithTx(ServiceConfigConverter.java:63) ~[?:?]
          ... 20 more

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

• ServiceConfig table in the NSX Manager Corfu DB has a stale reference to a Logical Router ID, which no longer exists.
• The Logical Router was deleted, but a related child attribute, which is a service profile (Flood Protection profile) in the ServiceConfig table was not deleted.

Resolution

This issue is resolved in VMware NSX 4.2.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Use the API to delete the flood-protection-profile-bindings and recreate it. Please open a NSX support ticket referencing this KB to get guidance on the workaround steps.