API queries to Orchestrator (VCO) fails or takes longer time

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

API queries to Orchestrator fails or takes longer time for shared VCOs running software version 6.2.x.x

Below error messages can be seen by customers while requesting API queries

{"jsonrpc":"2.0","error":{"code":-32603,"message":"Unknown error occurred"}}

Additional error codes  502 and 504 also seen.

Environment

Velocloud SD-WAN

Cause

In multi tenant (shared) VCOs, it is observed that some of the Enterprise API users are sending huge number of API queries frequently which causes CPU resources to go high.
This affects other Enterprise API users while waiting for their API responses.

Resolution

Short term resolution:

Avoid using below mentioned CPU intensive APIs frequently. Rather increase the API polling interval.

edge/getEdgeSDWANPeers

 

Long term resolution :

Engineering team is working to resolve the issue. The long term solution contains 3 step approach.

1. Enterprise specific API rate limiting
2. Reducing API response time
3. Optimizing the database for faster query processing

Additional Information

If you still experience issues with your Orchestrators , please reach out to Broadcom Support for more information