Events for Office 365 utilize high bandwidth resources causing delayed results

Products

• CASB Security Premium
• CASB Security Advanced
• CASB Security Standard

Issue/Introduction

CloudSOC and External DLP events for Onedrive and Sharepoint activities are delayed several hours before showing up as an Activity or Incident.

High latency is causing strain on bandwidth resources and take longer than 6 hours to be processed. 

Error 429
Error TOO MANY REQUESTS 

Environment

CASB/CloudSOC 2.96.1-1p

Cause

Rate limits are generally expected during the initial scanning as the Securlet is issuing multiple API calls simultaneously to process documents/sites from the SaaS.

Additionally, Microsoft began throttling API traffic in 2017 as outlined in MS Blog and the throttling was recently increased this year.

• This communication throttling increases latency and causes the processes of CloudSOC to take longer and throw the errors with Microsoft Message Center and other applications.

Note: CloudSOC is working as designed, for it is pulling the API's as configured from the cloud applications, but is being throttled on the application side.
 

Resolution

Confirmed Microsoft upgraded application server farms to send load based rate limits and lifted some restrictions they put in place while the Symantec Development Team made optimization changes within CloudSOC processes to help prevent these throttling delays.

Additional solutions proved to help relieve latency in some environments:

1. Review the following with Microsoft:
   • Verify there are no MSFT throttling restrictions based on license expiration etc.
   • Verify that other API activities are not occurring and utilizing the bandwidth of CloudSOC API calls
   • Review other products/features that make API calls within the environment
      
2. Deactivate the Securlet and re-activate with Selective Scanning:
   • Scoped by users/sites etc.
   • Allow for that initial scan to complete.