Large Cloudsoc deployments generate a large number of events, At times the high rate of events pushes the SIEM Agent polling rate to its limits which may cause the logs collection to lag behind.
As of the writing of this article, the Cloudsoc SIEM Agent polling rate is in the range of 25K-26K records per call. Large Cloudsoc tenants sometimes exceed this limit in the peak hours which reduces the polling capacity.
Any Cloudsoc environment that generates a large number of events and planning to poll all the events to their monitoring and logging systems.
To overcome the polling limits, multiple SIEM Agent instances can be configured in a way so that each one of the polls has a different set of data while collectively they poll all the events. This can be achieved using the "filtering" capabilities of the SIEM Agent.
The filtering can be done using the command line switches (SIEM Agent Reference):
Here is an example based on the first switch "Severity" :
1- Install Two (or multiple) instances of the SIEM Agent, each instance to have its own API Key [SIEM Agent Installation guide]
2- Run each SIEM Agent instance with a different set of Severities, so that collectively they poll all the events, for example:
the first instance can poll the events of severity "high" and "critical" like
splunk_agent.py --severity high critical --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP
while the second instance can poll the events of the rest of the severities like:
splunk_agent.py --severity error warning informational medium low --rate 5000 --output REMOTE --target localhost:6789 --socket_type TCP
3- configure a cron job for each instance to poll the event periodically (tweak depending on the requirement).
make sure to set the "rate" switch properly to reduce the polling time , in the above example a rate of 5K records per second was used "--rate 5000" , more details on this switch can be found on this article: [Increase rate limit]