NSX UI 'Search Index Out of Sync' Error (Code 513002) due to OpenSearch Memory Leak

Products

• VMware NSX

Issue/Introduction

• NSX User Interface (UI) fails to load inventory details.
• When attempting to view these details, one or more of the following error messages are displayed in the UI:
  
  
• • Error: Failed to fetch Inventory details. Please contact the administrator. Error: Search index is out of sync for 'Group'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. (Error code: 513002)


  • Failed to get report - Index out of sync. Please perform a re-sync.
    
    
  • Failed to get "Top Services by Distributed Firewall Rules" report - Index out of sync. Please perform a re-sync.
    
    
  • Failed to get "Distributed FW Rules" report - Index out of sync. Please perform a re-sync
    
    
  • Failed to get "IP ADDRESS MANAGEMENT" report - Index out of sync. Please perform a re-sync.
    
    
  • Failed to get "CONNECTIVITY" report - Index out of sync. Please perform a re-sync.
    
    
  • Failed to get "NETWORK SERVICES" report - Index out of sync. Please perform a re-sync.
    
    

• The OpenSearch logs (/var/log/search/opensearch.log) typically show repeated attempts to trigger Garbage Collection (GC) with little success:

[INFO ][o.o.i.b.HierarchyCircuitBreakerService] [nsx_node] attempting to trigger G1GC due to high heap usage [1918747568]
[INFO ][o.o.i.b.HierarchyCircuitBreakerService] [nsx_node] GC did bring memory usage down, before [1918747568], after [1822748288], allocations [1], duration [5]

• A core dump file by the name "search_oom.hprof" may be generated under "/image/core" on the manager node.
• This issue may occur in both federated and non federated environments.
• Within a federated NSX installation, this may be seen when selecting one or more of the local managers from the site-switcher drop down or when accessing one of the local managers directly without going via the global manager.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The issue is caused by the OpenSearch service running out of memory (OOM) and crashing, which causes the search index to fall out of sync with the NSX database.

This is due to a known memory leak in the Jackson library (version 2.17.0), which is bundled with OpenSearch 2.13.0.

Resolution

This is a known issue impacting VMware NSX 4.2.1.X, 4.2.2.X and 4.2.3.X. This issue is resolved in VMware NSX 9.0 and higher, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Restart the search service on the affected NSX Manager node(s) to clear the memory and re-initialize the service using the below command (run as the admin user):

> restart service search

After the search service is restarted, run start search resync all on all NSX manager nodes in the affected cluster.