Defining the rate at which queued real time events are uploaded to the Data Center Security Manager

book

Article ID: 181626

calendar_today

Updated On:

Products

Critical System Protection

Issue/Introduction

 

Resolution

Background:
All events generated in an agent is stored in the SISIDSEvents.csv file. Events
that are configured by the user to be sent to the server in real-time are copied
to the SISRTEvents.csv.

Real-Time Event Transmission:
The agent continuously monitors (every 1 second) SISRTEvents.csv file  to
determine if there are events available that needs to be sent to the server. The
agent remembers the last event it has successfully sent to the server. The agent
performs processing on newer events and then sends a set of events to the
server. One of the processing tasks that the agent does is - Consolidation. If
an event occurs consecutively the agent will try to consolidate that set of
events to a single event before it sends to the server. This is way to minimize
transmitting the same kind of events that are sent to the server. These actions
are controlled by the reader limit and the writer limit and log consolidation
settings (Prevention Config).

Reader Limit: The reader limit is the maximum number of events processed before
the agent pauses. By pausing after a specified number of events, the agent
avoids consuming too many system resources. Increasing the reader limit lets the
agent consume more resources, but gets events to the Monitors page more quickly.
Decreasing the reader limit reduces the resources that the agent consumes. This
setting will not affect the agent network transmission directly.

Writer Limit: The writer limit is the maximum number of events that the agent
can send in a single TCP/IP connection. Creating a TCP/IP connection is
overhead, and sending multiple events in a single connection reduces the average
overhead per event.

In every time interval, the agent starts reading and processing of the events.
The maximum number of events it will read and process depends on the Reader
limit. If the agent hits the writer limit during reading and processing of the
events, it will stop reading anymore events from the file. It will then create a
TCP/IP session to send this set of events. It will wait for 1 second, before
reading the next set of events based on the reader limit.

Additionally, not all events are of the same size. Sizes of events will vary
greatly based on the type of event. 

So as we can see from a proper determination of sending events to the server,
the following set of conditions impact the number -

1. Reader Limit
2. Writer Limit
3. Consolidation Settings
4. Event Size  (maximum size of 10KB)
5. Time required to transmit each packet (agent waits for a maximum of 5 minutes
before it fails sending)

A lower Writer limit will create a small set of events to be transmitted while a
higher Writer limit will send a bigger set of events. The optimal Writer limit
will depend on the network bandwidth available.


Now to answer specific customer questions:
  Question: What causes the agent to brake the session?

1.      The queue is empty.
2.      The Writer limit is reached.
3.    Network Connectivity goes down.

  Question: What causes the agent to start the TCP-IP session?

1.    Resumption of Network Connectivity.    
2.      Queued events ready for upload.

Note: There is no relation between agent's heartbeat interval and real-time
event transmission to the server.

  Question: What does the writer limit define is it per second, or some other
unit of time?

A writer limit determines the number of events in a queue that will be part of a
TCP/IP session sent to the server. Since the queue is created every second after
the last transmission of the events, we can say that the writer limit is
approximately equal to number of events per second.
However, we should note the following exceptions :-
1.  If the reader limit is pretty high and a lot of log consolidation happens,
this limit maybe more than a second.
2. On slower network connection or low bandwidth networks, we might see
transmission time to send a batch of events increase. This means that the writer
limit may not necessarily correspond to a number of events per second.

Bulk-Log Transmission:
Bulk-log transmission is independent of real-time log transmission. The agent
monitors the scsplog folder every 10 minutes. If it finds a single  SISIDSEvents
file rolled over, it will zip it and store it in the upload directory. it will
then attempt to upload it to the server in a single TCP-IP session. If it is
unable to upload it, it will retry to send it after 1 minute. If there are
multiple files that are rolled over, the agent will process each file and zip it
every 1 minute. And it will then upload the files every 1 minute until all the
files are transmitted. 

Note:
1. Both Bulk-log or Real Time Event transmission cannot happen together. If one
is happening, the other will wait for it to end before it starts.  
2. For slow networks, time required to transmit each packet becomes very
important. A higher writer limit may not guarantee a faster sending of events.
3. For low bandwidth networks, size of packets become important too.

Every event is different. Detection events generally will have a larger size
than prevention events. Virtual events may be a larger size than detection
events. But if I have to really comment on an average event size it would be
between 2 to 5KB.
Writer limit is equivalent to Per second only if the network has large bandwidth
and fast enough. If the network is slow or low bandwidth, the writer limit will
no longer be equivalent to per second.

Default Writer Limit is 10

More answers to customer questions:

For all the calculations below, I am trying to provide the scenario where time
required to transmit each session is negligible. However, the whole purpose as I
see it Standard Chartered is to figure out what optimal message transmission
rate should be for low bandwidth/slow WAN networks. We cannot provide guidance
for that. The Writer Limit is the control they have in conjunction with the
policy they apply and a careful understanding of events they want to filter out
and send it to the server.


As mentioned previously - the following set of conditions impact the
transmission rate -

1. Reader Limit
2. Writer Limit
3. Consolidation Settings
4. Event Size  (maximum size of 10KB)
5. Time required to transmit each packet (agent waits for a maximum of 5 minutes
before it fails sending)


Question: I am trying to define the rate at which queued real time events will
be uploaded to the manager after it has been unreachable for an extended period
of time. With the default setting of 1,000 reader limit and the 10 writer limit
what would the network utilization be for 200 megabytes of queued up events take
to upload on a wide open gigabit connection?


For the purposes of this sizing, let’s assume the event size to be flat 10KB,
and there is no log consolidation.

10 Writer Limit = 10 events per TCP/IP session = 10*10KB = 100KB data per batch
(per second). 6MB in a minute
This means, we are looking at an approximate number of 2000 TCP/IP sessions.

If we assume the time required to transmit each session is negligible, it would
take approximately 33 minutes.
Note: If the average event size reduces from 10KB, the amount of data
transmitted per session will be less.

Question: With a reader limit of 10,000 and a writer limit of 1 what is the
network utilization for 200 Megabytes of queued up events baring any other
overhead  a wide open gigabit connection?


For the purposes of this sizing, let’s assumethe event size to be flat 10KB, and
there is no log consolidation.

1 Writer Limit = 1 event per TCP/IP session =  10KB = 10KB data per batch (per
second). 0.6MB in a minute
This means, we are looking at an approximate number of 20000 TCP/IP sessions.

If we assume the time required to transmit each session is negligible, it would
take approximately 333 minutes or almost 6 hrs.
Note: This may reduce if log consolidation is happening.

Question:Events in the database are about 2 kilobytes in size so the default
1,000 writer limit would make for about 16 Megabits of network traffic not
accounting for TCP-IP overhead per session correct?


If we assume the time required to transmit each session is negligible,
1000 Writer Limit = 1000 events per TCP/IP session = 1000*2KB = 1.95MB data per
batch (per second).