SIEM Agent in large deployments and the way to Increase the polling capacity

Products

• CASB Audit
• CASB Security Advanced
• CASB Security Standard
• CASB Security Premium

Issue/Introduction

Large Cloudsoc deployments generate a large number of events. At times the high rate of events pushes the SIEM Agent polling rate to its limits which may cause the logs collection to lag behind.

 

Environment

This KB applies to any Cloudsoc environment that generates a large amount of events and plans to poll all those events to their monitoring and logging systems.

Cause

As of the writing of this article, the Cloudsoc SIEM Agent polling rate is in the range of 25K-26K records per call.  Large Cloudsoc tenants sometimes exceed this limit in peak hours, reducing the polling capacity. 

Resolution

To overcome the polling limits, multiple SIEM Agent instances can be configured in a way so that each one of the polls has a different set of data. Collectively all the SIEM agents poll all the events. This can be achieved using the "filtering" capabilities of the SIEM Agent.

The filtering can be done using the command line switches (SIEM Agent Reference):

• "Severity" : this switch is used to poll the events of certain severity(ies)
• "Object_Type" : this switch  is used to poll the events of a certain object type(s)

 

Here is an example based on the first switch "Severity" : 

1- Install Two (or multiple) instances of the SIEM Agent, each instance to have its own API Key [SIEM Agent Installation guide]

2- Run each SIEM Agent instance with a different set of Severities, so that collectively they poll all the events, for example:

 

the first instance can poll the events of severity "high" and "critical" like

splunk_agent.py --severity high critical --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP

 

while the second instance can poll the events of the rest of the severities like:

splunk_agent.py --severity error warning informational medium low --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP

 

3- configure a cron job for each instance to poll the event periodically (tweak depending on the requirement).

 

Additional Information

Please make sure to set the "rate" switch properly to reduce the polling time. In the above example a rate of 5K records per second was used "--rate 5000", more details on this switch can be found on this article:  [Increase rate limit]

Please note that since we use syslog as the forwarding mechanism, it's not possible to run two agents on the same system.