Configure "The Number Of Log Forwarders" increased the limit of forwarders up to 20 per cluster.
Article ID: 376529
Updated On:
Products
Issue/Introduction
VMware Aria Operations for Logs used to have a limit of 10 forwarders.
Starting 8.18 GA 10 is the default allowed count but the admin can increase the limit of forwarders up to 20 per cluster.
Changing the default value of limit means using the PATCH /api/v2/limits/max-log-forwarder-count API./
The API requires this body:
{
"value": "20"
}
The recommended maximum of log forwarders per Log Insight instance is 20. Configurations having more than 20 log forwarders are unsupported.
Environment
Aria Operations for Logs 8.18.x
Cause
Each forwarder can affect the performance of the cluster and has the potential to impact the ingestion rate. It was observed that there are cases where complex forwarders affect ingestion rate and causes accumulation of disk blocks.
Resolution
The following configurations have been tested internally to identify the impact on the ingestion rate and the recommendations.
The testing were done on a 3 large node cluster with 18000 events per second ingestion.
The below scenarios do not impact the ingestion negatively. Consider them when configuring forwarders:
Configuring 5 forwarders with 8 filters which use extracted fields.
Example:
text matches *error*
text matches *user*
text matches *time*
text matches *config*
text matches *is*
text matches *set*
text matches *correct*
text matches *review*
The following configurations have been tested internally to identify the impact on the ingestion rate and the recommendations.
The testing were done on a 3 large node cluster with 18000 events per second ingestion.
The below scenarios do not impact the ingestion negatively. Consider them when configuring forwarders:
Configuring 5 forwarders with 8 filters which use extracted fields
Example:
text matches *error*
text matches *user*
text matches *time*
text matches *config*
text matches *is*
text matches *set*
text matches *correct*
text matches *review*
The following configurations were tested and documented as reference.
Configuration 1 - Configure 20 forwarders with no filter (fwd all logs) + and complementary tags enabled
|
Data Name
|
Before Configuring Forwarders
|
After Configuring Forwarder
|
|---|---|---|
| CFAPI Events Incoming Rate (Per Second) → Last Five Minutes | 15,197 | 15,188 |
| Disk Blocks | 0 | 1 |
| Events Ingestion Rate (Per Second) → Last Five Minutes | 56,185 | 56,405 |
| Events Ingestion Volume (MBs Per Second) → Last Five Minutes | 21.03 | 21.1 |
| Syslog Events Incoming Rate (Per Second) → Last Five Minutes | 40,993 | 40,995 |
Configuration 2 - Configure 10 forwarders with 8 filters which use extracted fields, cfapi, SSL on
|
Data Name
|
Before Configuring Forwarders
|
After Configuring Forwarder
|
|---|---|---|
| Events Ingestion Rate (Per Second) → Last Five Minutes | 56,067 | 56,285 |
| Events Ingestion Volume (MBs Per Second) → Last Five Minutes | 21.05 | 21.07 |
| Syslog Events Incoming Rate (Per Second) → Last Five Minutes | 40,999 | 41.002 |
| CFAPI Events Incoming Rate (Per Second) → Last Five Minutes | 15,187 | 15,190 |
| Disk Blocks | 0 | 2 |
Configuration 3 - Configure 10 forwarders with 8 filters which use extracted fields, syslog, SSL on
|
Data Name
|
Before Configuring Forwarders
|
After Configuring Forwarder
|
|---|---|---|
| Events Ingestion Rate (Per Second) → Last Five Minutes | 56,285 | 56,289 |
| Events Ingestion Volume (MBs Per Second) → Last Five Minutes | 21.07 | 21.04 |
| Syslog Events Incoming Rate (Per Second) → Last Five Minutes | 41.002 | 40,997 |
| CFAPI Events Incoming Rate (Per Second) → Last Five Minutes | 15,190 | 15,180 |
| Disk Blocks | 2 | 3 |
Configuration 4 - Configure 20 forwarders with 8 filters which use extracted fields, syslog, SSL on
|
Data Name
|
Before Configuring Forwarders
|
After Configuring Forwarder
|
|---|---|---|
| Events Ingestion Rate (Per Second) → Last Five Minutes | 56,289 | 56,087 |
| Events Ingestion Volume (MBs Per Second) → Last Five Minutes | 21.04 | 21.04 |
| Syslog Events Incoming Rate (Per Second) → Last Five Minutes | 40,997 | 41,003 |
| CFAPI Events Incoming Rate (Per Second) → Last Five Minutes | 15,180 | 15,205 |
| Disk Blocks | 3 | 11 |
Note: CPU usage on one of the nodes spiked.
Configuration 5 - Configure 20 forwarders with 8 filters which use extracted fields, cfapi, SSL on
|
Data Name
|
Before Configuring Forwarder
|
After Configuring Forwarder
|
|---|---|---|
| Events Ingestion Rate (Per Second) → Last Five Minutes | 56,087 | 56,416 |
| Events Ingestion Volume (MBs Per Second) → Last Five Minutes | 21.04 | 21.2 |
| Syslog Events Incoming Rate (Per Second) → Last Five Minutes | 41,003 | 40,995 |
| CFAPI Events Incoming Rate (Per Second) → Last Five Minutes | 15,205 | 15,217 |
| Disk Blocks | 11 | 12 |
Note: CPU usage on one of the nodes spiked.
Additional Information
Recommendation :
In case if ingestion decreases or if disk blocks are noted, vertical or horizontal scaling out/up is suggested. Disk blocks is a mechanism which Log Insight leverages in cases when it is not able to ingest the incoming traffic in real time and is forced to store the logs to disk for a later processing.
Feedback
