Events for Office 365 utilize high bandwidth resources causing delayed results

book

Article ID: 171304

calendar_today

Updated On:

Products

CASB Security Premium

Issue/Introduction

Found CloudSOC and External DLP events for Onedrive and Sharepoint activities are delayed several hours before showing up as an Activity or Incident.

Found this high latency is causing strain on bandwidth resources and take longer than 6 hours to be processed. 

Error 429
Error TOO MANY REQUESTS

Cause

Rate limits are generally expected during the initial scanning as the Securlet is issuing multiple API calls simultaneously to process documents/sites from the SaaS.

Additionally, Microsoft began throttling API traffic in 2017 as outlined in MS Blog and the throttling was recently increased this year.

  • This communication throttling increases latency and causes the processes of CloudSOC to take longer and throw the errors with Microsoft Message Center and other applications.

Note: CloudSOC is working as designed, for it is pulling the API's as configured from the cloud applications, but is being throttled on the application side.
 

Environment

CASB/CloudSOC 2.96.1-1p

Resolution

Confirmed Microsoft upgraded application server farms to send load based rate limits and lifted some restrictions they put in place while the Symantec Development Team made optimization changes within CloudSOC processes to help prevent these throttling delays.

Additional solutions proved to help relieve latency in some environments:

  1. Review the following with Microsoft:
    • Verify there are no MSFT throttling restrictions based on license expiration etc.
    • Verify that other API activities are not occurring and utilizing the bandwidth of CloudSOC API calls
    • Review other products/features that make API calls within the environment
       
  2. Deactivate the Securlet and re-activate with Selective Scanning:
    • Scoped by users/sites etc.
    • Allow for that initial scan to complete and add tot he scope