CloudSOC Importer submits too many API requests

Products

• Information Centric Analytics

Issue/Introduction

The CloudSOC importer job fails and logs error code 99 in the SQL Server Agent history. The importer log shows HTTP Status Code 429 instead ('Too many requests'):

2023-06-22 08:02:37,404 JsonImporter.exe [1:INFO] ApiClient.GetResponse() Calling https://api-vip.elastica.net/<tenant-name>/audit/v2/usernames/
2023-06-22 08:02:37,467 JsonImporter.exe [1:ERROR] ApiClient.GetResponse() Http Status Code was 429. Response saved to E:\Program Files\Bay Dynamics\Database Utilities\JsonImporter_1.3.1.0_signed\Error_AuditUsernames_20230622080237_wfkymw5s.dc5.json.
2023-06-22 08:02:37,467 JsonImporter.exe [1:ERROR] ApiClient.GetResponse() received 429 Could not retrieve data. Status code 429.
System.Exception: Could not retrieve data. Status code 429.
   at JsonImporterUtils.ApiClient.GetResponse(Endpoint endpoint, IRequest request, Func`4 fParseAction)

The response file referenced in the error only contains the string 'Too many requests'.

Environment

Release : 1.3.1.0

Component : CloudSOC Importer

Cause

The CloudSOC importer is failing due to CloudSOC's API polling rate limit. From the CloudSOC API guide:

429

Too many requests

The CASB Audit API is unable to service the client until a reasonable amount of time has elapsed. We expect the default wait time to be approximately 1 minute. The Retry-After response header displays the remaining wait time in seconds.

https://techdocs.broadcom.com/us/en/symantec-security-software/information-security/symantec-cloudsoc/cloud/api-home/audit-api/audit-response-codes.html

NOTE: Link last validated July 6, 2023

Resolution

The CloudSOC Importer will be updated in a future release to dynamically throttle its polling rate in response to the CloudSOC API's rate limits. For assistance with this issue, contact Broadcom Support.