ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

SIEM Agent in large deployments and the way to Increase the polling capacity

book

Article ID: 223370

calendar_today

Updated On:

Products

CASB Audit

Issue/Introduction

Large Cloudsoc deployments generate a large number of events, At times the high rate of events pushes the SIEM Agent polling rate to its limits which may cause the logs collection to lag behind.

 

Cause

As of the writing of this article, the Cloudsoc SIEM Agent polling rate is in the range of 25K-26K records per call.  Large Cloudsoc tenants sometimes exceed this limit in the peak hours which reduces the polling capacity. 

Environment

Any Cloudsoc environment that generates a large number of events and planning to poll all the events to their monitoring and logging systems.

Resolution

To overcome the polling limits, multiple SIEM Agent instances can be configured in a way so that each one of the polls has a different set of data while collectively they poll all the events. This can be achieved using the "filtering" capabilities of the SIEM Agent.

The filtering can be done using the command line switches (SIEM Agent Reference):

  • "Severity" : this switch is used to poll the events of certain severity(ies)
  • "Object_Type" : this switch  is used to poll the events of a certain object type(s)

 

Here is an example based on the first switch "Severity" : 

1- Install Two (or multiple) instances of the SIEM Agent, each instance to have its own API Key [SIEM Agent Installation guide]

2- Run each SIEM Agent instance with a different set of Severities, so that collectively they poll all the events, for example:

 

the first instance can poll the events of severity "high" and "critical" like

splunk_agent.py --severity high critical --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP

 

while the second instance can poll the events of the rest of the severities like:

splunk_agent.py --severity error warning informational medium low --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP

 

3- configure a cron job for each instance to poll the event periodically (tweak depending on the requirement).

 

Additional Information

make sure to set the "rate" switch properly to reduce the polling time , in the above example a rate of 5K records per second was used "--rate 5000" , more details on this switch can be found on this article:  [Increase rate limit]