NSX IP Pool UI Error: Exception while searching during pipeline execution (Error code: 60576) - VMware NSX
Article ID: 454094
Updated On:
Products
Issue/Introduction
- When navigating to 'Networking > IP Address Pools > IP Address Pools' in the NSX Manager UI, the page fails to load, and the following error is displayed:
Error: Exception while searching during pipeline execution (Error code: 60576)
- Log analysis reveals the following errors in the NSX Manager logs:
- In the /var/log/proton/nsxapi.log of the NSX manager, there are a great many (hundreds) of these, repeating every 5 minutes:
INFO INTENT-PROCESSOR-CONSOLIDATED-SERVICE-1 ConsolidatedRealizedStateServiceImpl 5991 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updated consolidated state for intentPath:/infra/ip-pools/[IP-POOL-PATH]/ip-allocations/[ALLOCATION-ID] to:ERROR
- In the /var/log/proton/nsxapi.log of the NSX manager, there are a great many (hundreds) of these, repeating every 5 minutes:
- The following API shows the pool mentioned in the above log is exhausted:
GET /policy/api/v1/infra/ip-pools/[IP-POOL-PATH] - In the /var/log/search/search-manager.log of the NSX manager:
ERROR http-nio-127.0.0.1-7440-exec-6 SearchServiceImpl 5991 - [nsx@6876 comp="nsx-manager" errorCode="MP60506" level="ERROR" reqId="########-c9d8-4e5b-a36b-############" subcomp="manager" username="######@#######"] [QueryExecution: Execution] Exception while searching, query: resource_type:(IpAddressAllocation) java.io.IOException: entity content is too long [107740867] for the configured buffer limit [104857600]
- The following API shows the pool mentioned in the above log is exhausted:
Environment
VMware NSX
Cause
This error occurs when the OpenSearch buffer limit is exhausted. This is typically triggered by an accumulation of stale or persistent 'IpAddressAllocation' objects in an `ERROR` state.
These objects often accumulate due to repeated, automated allocation attempts (e.g., via Terraform or other orchestration tools) against an exhausted IP address pool. These allocation intents persist indefinitely, this leads to the resulting search payload size exceeding the configured buffer limit (100MB), causing the UI to fail when attempting to fetch the IP Pool data.
Resolution
To resolve this issue, you must identify and remove the stale/failed allocation intents.
- Identify Stale Allocations:
- Review the environment for automated orchestration loops (e.g., Terraform or other) that are repeatedly attempting to allocate IP addresses from an exhausted IP Address Pool.
- Clean Up Failed Allocations: Remove the persistent 'IpAddressAllocation' objects that are in the 'ERROR' state.
- This API can be used to identify the IP Pool(s): GET /policy/api/v1/infra/ip-pools
- Then check for the allocations of the Pool(s): GET /policy/api/v1/infra/ip-pools/<Pool_ID>/ip-allocations
Or - If you already know the allocation ID: GET /policy/api/v1/infra/realized-state/realized-entities?intent_path=/infra/ip-pools/<Pool_ID>/ip-allocations/<Allocation_ID>
- Check for allocations with "error_code" : 5109
- Then delete the failed allocation: DELETE /policy/api/v1/infra/ip-pools/<Pool_ID>/ip-allocations/<Allocation_ID>
- Prevent Recurrence:
- Ensure that the automation platform (e.g., Terraform or other) includes error handling to stop attempts when an exhaustion error (5109) is received.
- Consider increasing the IP address pool capacity if the demand is legitimate, or implementing cleanup logic within your automation scripts to handle failed allocation lifecycle events.
Once the stale 'IpAddressAllocation' objects are removed and the buffer size is no longer exceeded, the IP Address Pool UI should automatically become responsive again.
Known Issue and Updates
This behavior is a known issue. To stay informed about fix status or permanent remediations, please subscribe to this knowledge article. You can learn more about how to track defect status and receive updates at Subscribe to a Broadcom knowledge article by article or product.
Additional Information
Feedback
