Configure "The Number Of Log Forwarders" increased the limit of forwarders up to 20 per cluster.
search cancel

Configure "The Number Of Log Forwarders" increased the limit of forwarders up to 20 per cluster.

book

Article ID: 376529

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

VMware Aria Operations for Logs used to have a limit of 10 forwarders.

Starting 8.18 GA 10 is the default allowed count but the admin can increase the limit of forwarders up to 20 per cluster.

Changing the default value of limit means using the PATCH /api/v2/limits/max-log-forwarder-count API./

The API requires this body:

{
  "value": "20"
}

The recommended maximum of log forwarders per Log Insight instance is 20. Configurations having more than 20 log forwarders are unsupported.

Environment

Aria Operations for Logs 8.18.x

Cause

Each forwarder can affect the performance of the cluster and has the potential to impact the ingestion rate. It was observed that there are cases where complex forwarders affect ingestion rate and causes accumulation of disk blocks.

Resolution

The following configurations have been tested internally to identify the impact on the ingestion rate and the recommendations.
The testing were done on a 3 large node cluster with 18000 events per second ingestion.

The below scenarios do not impact the ingestion negatively. Consider them when configuring forwarders:

Configuring 5 forwarders with 8 filters which use extracted fields.

Example:
text matches *error*
text matches *user*
text matches *time*
text matches *config*
text matches *is*
text matches *set*
text matches *correct*
text matches *review*

The following configurations have been tested internally to identify the impact on the ingestion rate and the recommendations.

The testing were done on a 3 large node cluster with 18000 events per second ingestion.

The below scenarios do not impact the ingestion negatively. Consider them when configuring forwarders:

Configuring 5 forwarders with 8 filters which use extracted fields

Example:
text matches *error*
text matches *user*
text matches *time*
text matches *config*
text matches *is*
text matches *set*
text matches *correct*
text matches *review*

The following configurations were tested and documented as reference.

Configuration 1 - Configure 20 forwarders with no filter (fwd all logs) + and complementary tags enabled

Data Name
Before Configuring Forwarders
After Configuring Forwarder
CFAPI Events Incoming Rate (Per Second) → Last Five Minutes 15,197 15,188
Disk Blocks 0 1
Events Ingestion Rate (Per Second) → Last Five Minutes 56,185 56,405
Events Ingestion Volume (MBs Per Second) → Last Five Minutes 21.03 21.1
Syslog Events Incoming Rate (Per Second) → Last Five Minutes 40,993 40,995

 

Configuration 2 - Configure 10 forwarders with 8 filters which use extracted fields, cfapi, SSL on

Data Name
Before Configuring Forwarders
After Configuring Forwarder
Events Ingestion Rate (Per Second) → Last Five Minutes 56,067 56,285
Events Ingestion Volume (MBs Per Second) → Last Five Minutes 21.05 21.07
Syslog Events Incoming Rate (Per Second) → Last Five Minutes 40,999 41.002
CFAPI Events Incoming Rate (Per Second) → Last Five Minutes 15,187 15,190
Disk Blocks 0 2


Configuration 3 - Configure 10 forwarders with 8 filters which use extracted fields, syslog, SSL on

Data Name
Before Configuring Forwarders
After Configuring Forwarder
Events Ingestion Rate (Per Second) → Last Five Minutes 56,285 56,289
Events Ingestion Volume (MBs Per Second) → Last Five Minutes 21.07 21.04
Syslog Events Incoming Rate (Per Second) → Last Five Minutes 41.002 40,997
CFAPI Events Incoming Rate (Per Second) → Last Five Minutes 15,190 15,180
Disk Blocks 2 3

 

Configuration 4 - Configure 20 forwarders with 8 filters which use extracted fields, syslog, SSL on

Data Name
Before Configuring Forwarders
After Configuring Forwarder
Events Ingestion Rate (Per Second) → Last Five Minutes 56,289 56,087
Events Ingestion Volume (MBs Per Second) → Last Five Minutes 21.04 21.04
Syslog Events Incoming Rate (Per Second) → Last Five Minutes 40,997 41,003
CFAPI Events Incoming Rate (Per Second) → Last Five Minutes 15,180 15,205
Disk Blocks 3 11

Note: CPU usage on one of the nodes spiked.


Configuration 5 - Configure 20 forwarders with 8 filters which use extracted fields, cfapi, SSL on

Data Name
Before Configuring Forwarder
After Configuring Forwarder
Events Ingestion Rate (Per Second) → Last Five Minutes 56,087 56,416
Events Ingestion Volume (MBs Per Second) → Last Five Minutes 21.04 21.2
Syslog Events Incoming Rate (Per Second) → Last Five Minutes 41,003 40,995
CFAPI Events Incoming Rate (Per Second) → Last Five Minutes 15,205 15,217
Disk Blocks 11 12

Note: CPU usage on one of the nodes spiked.

Additional Information

Recommendation :

In case if ingestion decreases or if disk blocks are noted, vertical or horizontal scaling out/up is suggested. Disk blocks is a mechanism which Log Insight leverages in cases when it is not able to ingest the incoming traffic in real time and is forced to store the logs to disk for a later processing.