NSX UI 'Search Index Out of Sync' Error (Code 513002) due to OpenSearch Memory Leak
search cancel

NSX UI 'Search Index Out of Sync' Error (Code 513002) due to OpenSearch Memory Leak

book

Article ID: 419175

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

This article outlines the symptoms, cause, and resolution for UI inventory loading failures and "Search index is out of sync" error messages within VMware NSX.

Symptoms:

  • NSX User Interface (UI) fails to load inventory details.

  • When attempting to view these details, one or more of the following error messages are displayed in the UI:

    • Error: Failed to fetch Inventory details. Please contact the administrator. Error: Search index is out of sync for 'Group'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. (Error code: 513002)

    • Failed to get report - Index out of sync. Please perform a re-sync.

    • Failed to get "Top Services by Distributed Firewall Rules" report - Index out of sync. Please perform a re-sync.

    • Failed to get "Distributed FW Rules" report - Index out of sync. Please perform a re-sync

    • Failed to get "IP ADDRESS MANAGEMENT" report - Index out of sync. Please perform a re-sync.

    • Failed to get "CONNECTIVITY" report - Index out of sync. Please perform a re-sync.

    • Failed to get "NETWORK SERVICES" report - Index out of sync. Please perform a re-sync.

  • The OpenSearch logs (/var/log/search/opensearch.log) typically show repeated attempts to trigger Garbage Collection (GC) with little success:

[INFO ][o.o.i.b.HierarchyCircuitBreakerService] [nsx_node] attempting to trigger G1GC due to high heap usage [1918747568]
[INFO ][o.o.i.b.HierarchyCircuitBreakerService] [nsx_node] GC did bring memory usage down, before [1918747568], after [1822748288], allocations [1], duration [5]

  • A core dump file by the name "search_oom.hprof" may be generated under "/image/core" on the manager node.

  • This issue may occur in both federated and non federated environments.

  • Within a federated NSX installation, this may be seen when selecting one or more of the local managers from the site-switcher drop down or when accessing one of the local managers directly without going via the global manager.

  • Proton and CM-Inventory OOM: The OpenSearch memory leak can lead to proton and cm-inventory service Out-of-Memory (OOM) events across all three manager nodes. /image/core/proton_oom.hprof & /image/core/cm_inventory_oom.hprof
  • Specific Error Code: MP60525: [2026-07-01T17:25:03.267Z ERROR UfoReIndexingManager-Reindexing-0 UfoReIndexingManager 77235 - [nsx@6876 comp="nsx-manager" errorCode="MP60525" level="ERROR" subcomp="manager"] Indexing: Reindexing] Unrecoverable error during re-indexing and subscribing. Search framework initialization failed. org.opensearch.OpenSearchStatusException: OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large...]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

A known memory leak in the Jackson library (version 2.17.0) bundled with OpenSearch 2.13.0 progressively consumes available heap, eventually causing the OpenSearch service to run out of memory and crash. When this occurs, the search index falls out of sync with the NSX database.

The reduced heap headroom also has a secondary effect. When a start search resync all or a standard indexing request is triggered, OpenSearch rejects it via the circuit breaker, prompting a high-frequency retry storm from the search service. The resulting accumulation of sync data can drive further OOM failures in the proton and cm-inventory services across all manager nodes in the cluster.

Resolution

This is a known issue impacting VMware NSX 4.2.1.X, 4.2.2.X and 4.2.3.X.  

The issue is resolved in VMware NSX 4.2.4 (for 4.X) and 9.0 and higher (for 9.X). Upgrade to these versions to permanently address the memory leak available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Restart the search service on the affected NSX Manager node(s) to clear the memory and re-initialize the service using the below command (run as the admin user):

  • nsxmgr> restart service search

After the search service is restarted, run start search resync all on all NSX manager nodes in the affected cluster.

Verify the status of proton and cm-inventory services, as they may also require restarts if they were impacted by the retry storm:

  • nsxmgr> get service manager
    • nsxmgr> restart service manager
  • nsxmgr> get service cm-inventory
    • nsxmgr> restart service cm-inventory

Alternatively, a rolling reboot of the 3 NSX Manager VMs will address the above service restarts:

  1. Reboot the first NSX Manager (Non-VIP Leader).
  2. SSH to a different NSX Manager VM as admin user and check cluster health: get cluster status
  3. When all services report up on all 3 NSX Manager nodes, reboot the next NSX Manager (Non-VIP Leader).
  4. Repeat steps 2-3 for the last NSX Manager (VIP leader).

Additional Information

OpenSearch のメモリリークによる NSX UI 上の 'Search Index Out of Sync' エラー (Code 513002)

Powered by Wolken Software