NSX Manager search fails with an error - "Error: Search index is out of sync. Run the 'start search resync all'
search cancel

NSX Manager search fails with an error - "Error: Search index is out of sync. Run the 'start search resync all'

book

Article ID: 369502

calendar_today

Updated On:

Products

VMware NSX VMware vDefend Firewall

Issue/Introduction

  • Service config resource refers to Logical Router UUID, which no longer exists in the system. This may cause exceptions in the UI search. 
    "Error: Search index is out of sync. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [all] (Error code: 60516)"



  • While reviewing the NSX Manager logs, the user will observe "Search index is out of sync 'resource_type:ServiceConfig'."
 /var/log/syslog

2024-03-29T19:59:33.782Z <NSX_Manager_Name> NSX 846255 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'nsx-search' Operation:'GET@/api/v1/search/query' Operation status: 'failure' Error: Search index is out of sync for 'resource_type:ServiceConfig'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support.

Environment

VMware NSX-T Data Center
VMware NSX

Vdefend Firewall

Cause

Global search returns error "Index out of sync, please resync." This is due to missing data in ServiceConfig table for associated logical routers. Search resync doesn't help because there is missing data in corfu which prevents DTO convertion for ServiceConfig entries.

It is caused under 2 conditions:

  1. Create a T1/T0 logical router and attach flood-protection profile to it. Delete T1/T0. This leaves flood-protection profile as stale meaning which a T1/T0 logical router was deleted sometime back and has never been created back with same T1/T0 name. Flood-protection profile will reference to old UUID. 
  2. Create a T1/T0 logical router and attach flood-protection profile to it. Delete T1/T0. We now recreate T1/T0 logical router with the same T1/T0 name. This also leaves flood-protection profile as stale. Flood-protection profile will reference to old UUID. 
/var/log/search-manager.log

2024-03-29T19:59:33.782Z ERROR UfoIndexer-BatchExecutor-search_manager-1 UfoGenericConverter 4792 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='ServiceConfig', streamTag=MANAGER}, identifier=uuid {
 left: XXXXXXXXXXXXXXXXXXXX
 right: YYYYYYYYYYYYYYYYYYYY
}
}
Caused by: com.vmware.nsx.management.common.exceptions.ObjectNotFoundException
        at com.vmware.nsx.management.container.dao.IdentifiableProxyObjectDao.findByIdentifier(IdentifiableProxyObjectDao.java:400) ~[?:?]
        at com.vmware.nsx.management.edge.lrouter.service.LogicalRouterServiceImpl.getLogicalRouter(LogicalRouterServiceImpl.java:391) ~[?:?]
        at com.vmware.nsx.management.serviceconfig.converter.ServiceConfigConverter.toDtoWithTx(ServiceConfigConverter.java:219) ~[?:?]
        at com.vmware.nsx.management.serviceconfig.converter.ServiceConfigConverter.toDtoWithTx(ServiceConfigConverter.java:63) ~[?:?]
        ... 20 more

Resolution

This issue is resolved in VMware NSX 4.2.0

Workaround :

The resolution requires validation of NSX Manager corfu database tables. Please contact Broadcom support.

Creating and managing Broadcom support cases
 

 

Additional Information

  • This issue may affect the functionality of users trying to configure the AVI feature "Preserve Client IP"
    • During the creation/Deletion of service definitions the user may observe the "Search index out of sync" error in AVI and NSX logs
      • In the AVI controller logs the error will reference the service defintion to be used.
/var/lib/avi/log/cc_agent_go_####.log

2023-03-29T15:26:46.586Z        ERROR   nsxt/nsxt_si_task.go:69 [################################-ServiceDefinition] Search index is out of sync for 'resource_type:ServiceConfig'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support.
    • After running the command "start search resync all" on the NSX Managers with the error, the user may observe the exceptions found in the cause section of this KB.

 

Powered by Wolken Software