NSX UI is not loading, showing a "Search index is out of sync for Generic Policy realized Resource" error."

Products

• VMware NSX-T Data Center
• VMware NSX

Issue/Introduction

• If you have recently upgraded to VMware NSX version 4.1.1 and onwards. In the case where customer is deployed ALB Using LCM or non on boarding flow in 3.2.x
• The Networking part in the NSX UI stops working with the error: INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [all] (Error code: 60516).
• You've attempted the fix proposed in the error: "Run the 'start search resync all' CLI command on the NSX appliance to resync", and the error still exists. 
• In the /var/log/search/search-manager.logs file on an NSX-T manager, you see entries similar to:
• Following failures are occurring while indexing policy alarms or GenericPolicyRealizedResource

 

GPRR

2023-08-22T19:25:48.935Z  WARN pool-742-thread-1 GenericPolicyRealizedResourceDataProvider 105456 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] [IndexDataProvider-GenericPolicyRealizedResource] Could not fetch data to be indexed for GenericPolicyRealizedResource//infra/realized-state/enforcement-points/alb-endpoint/<ANY ALB Objects starts with alb_>/<name of the object>,

cause : 

com.vmware.nsx.management.policy.policyframework.exceptions.InvalidPolicyPathException:

The path=[/infra/realized-state/enforcement-points/alb-endpoint/<ANY ALB Objects starts with alb_>/<name of the object>] is invalid

 

 

2023-08-22T19:25:42.221Z ERROR pool-742-thread-1 UfoGenericConverter 105456 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to

Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='Alarm', streamTag=POLICY}, identifier=string_id: "/infra/realized-state/enforcement-points/alb-endpoint/<ANY ALB Objects starts with alb_>/<name of the object>/alarms/<UUID>" , timestamp.sequence =16398, timestamp.epoch=0}

java.lang.reflect.InvocationTargetException: null

 

• Apart from above issue You may also see similar entries to the below in the /var/log/proton/nsxapi.log saying

type.googleapis.com/vmware.nsx.ufostore.advanceloadbalancer.model.AlbControllerAdminCredsMsg not in map!"

Environment

VMware NSX-T Data Center
VMware NSX

Cause

• Issue is noticed during an upgrade from VMware NSX 3.2.x to 4.1.1 and onwards
• This issue occurs when some ALB objects remain in NSX-T DB.From VMware NSX 4.1.1 onwards, some ALB integration features were removed, this lead to some stale objects remaining and resulted in the above Generic Policy Realized Resource indexing Alarm above: InvalidPolicyPathException.
  Please refer to the following KB for more detail on the ALB deprecation: https://knowledge.broadcom.com/external/article/303357/deprecation-announcement-of-nsxt-alb-pol.html

Resolution

To resolve this, a few entries need to be cleaned up in the CORFU DB. Please contact Broadcom Support for guidance on the remediation process.

 

Additional Information

Kindly collect below table to remediate the issue 

• /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t ALBControllerAdminCreds >> albcreds.dump
• /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t GenericPolicyRealizedResource >> gprr.dump
• /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t Alarm >> alarm.dump