SIEM Agent importing speed is slower than the amount of new logs generated

Products

• CASB Security Advanced
• CASB Security Premium
• CASB Security Standard

Issue/Introduction

Splunk SIEM agent it's pulling around 26000 logs every run, but this is less than the amount of the new logs generated. As a result, the SIEM export is falling behind.

Cause

1. The Agent is trying to pull all CloudSOC logs, including some information that is not essential for exporting.
2. The gap between each Job is too big

Environment

Release : 1.0

Component :

Resolution

1. Use filters in the technote to import selective activity_type, severity, object_type can reduce the number of nonessential logs to be queried
   • python <tool> _agent.py [--proxy <host_and_port> ] [-u <username> -p <password> ] [--severity <severity ...> ] [--app <app ...> ] [--object_type <object_type ...> ] [--activity_type <activity_type ...> ] [--elastica_app <elastica_app ...> ] [-c] [-r] [-v] [-d] [--rate] [-o/--output] [--start_date <start_date> ] [-s/--stream <stream> ] [-t/--target <socket> ] [--socket_type <udp_or_tcp> ] [-f/--filename <filename> ] [--max_bytes <maximum_bytes> ] [--backup_count <backup_count> ]
2. Set up multiple agents with each agent pulls different Securlet logs using the --app option 
3. Check the time it takes to complete a job. Reduce the interval between jobs. For example, if a job takes 10 mins to complete, then adjust the schedule so the agent will run the job every 15 mins.
4. Check the log writing rate. The default rate is 40 logs per second, this can be increased to a maximum of 5000 logs per second.
   • For example, the logs below indicate it takes about 10 mins to write 25900.

YYYY-MM-DD 10:58:44,278-Log_Exporter_Client-INFO-Writing log to Syslog.

YYYY-MM-DD 11:09:40,112-Log_Exporter_Client-INFO-Wrote 25900 logs to Syslog.

• • This can be improved and adjusted so it will take less time to write the logs and therefore finish the job faster