Aria Operations for Logs generates a large number of DNS queries to DNS servers
search cancel

Aria Operations for Logs generates a large number of DNS queries to DNS servers

book

Article ID: 315938

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

  • There is a high number of DNS requests from Aria Operation for Logs to the internal DNS server(s), upwards of 50% of all queries

Environment

Aria Operations for Logs 8.x 

Resolution

This is a known issue affecting Aria Operations for Logs, there is no resolution at this time.  Please subscribe to this article to be informed when updates are published.

Workaround:
To workaround this issue, follow one of the beneath options for the ingestion source(s) of your choosing

Note: It is recommended to disable/adjust these features gradually, rather than all in one sitting.  For example, if you've gone through disabling/adjusting the syslog value and still feel displeased with the outcome, then proceed with adjusting the API server and Aria Operations integration.

Syslog Server

How to adjust:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config and under the following section, increase the dns-cache-entries value for syslog

Note: Replace vRealizeLogInsightIPorFQDN with the IP or Fully Qualified Domain Name (FQDN) of your environment.

<importer>
.
.
.
<dns-cache-entries value="102400" />
</importer>

Note: The value for dns-cache-entries parameter should be set to higher than the expected spikes in the number of hosts logging to Log Insight. There is currently no auto adjustment mechanism for this parameter.

  1. SSH to the primary node as root
Note: If you do not know your root password, refer to: How to reset the root password in Aria Operations for Logs (53649)
  1. Restart the service by running the following command
service loginsight restart
  1. Repeat steps 2-3 sequentially on all nodes in the Aria Operations for Logs cluster.

How to disable:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config and under the following section, set the syslog-resolve-source value value to false.

Note: Replace vRealizeLogInsightIPorFQDN with the IP or Fully Qualified Domain Name (FQDN) of your environment.

<import>
.
.
.
<syslog-resolve-source value="false" />
</import>

  1. SSH to the primary node as root

Note: If you do not know your root password, refer to: How to reset the root password in Aria Operations for Logs (53649)

  1. Restart the service by running the following command

service loginsight restart

  1. Repeat steps 2-3 sequentially on all nodes in the Aria Operations for Logs cluster.

API Server

How to adjust:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config and under the following section, increase the dns-cache-entries value for API:.

Note: Replace vRealizeLogInsightIPorFQDN with the IP or Fully Qualified Domain Name (FQDN) of your environment.

 <api-server>
.
.
.
<dns-cache-entries value="102400" />
</api-server>

Note: The value for dns-cache-entries parameter should be set to higher than the expected spikes in the number of hosts logging to Log Insight. There is currently no auto adjustment mechanism for this parameter.

  1. SSH to the primary node as root
Note: If you do not know your root password, refer to: How to reset the root password in Aria Operations for Logs (53649)
  1. Restart the service by running the following command
service loginsight restart
  1. Repeat steps 2-3 sequentially on all nodes in the Aria Operations for Logs cluster.


How to disable:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config and under the following section, set the api-resolve-source value to false.
Note: Replace vRealizeLogInsightIPorFQDN with the IP or Fully Qualified Domain Name (FQDN) of your environment.
<api-server>
.
.
.
<api-resolve-source value="false" />
</api-server>
  1. SSH to the primary node as root
Note: If you do not know your root password, refer to: How to reset the root password in Aria Operations for Logs (53649)
  1. Restart the service by running the following command
service loginsight restart
  1. Repeat steps 2-3 sequentially on all nodes in the Aria Operations for Logs cluster.

Aria Operations

How to adjust:

In the case where Aria Operations integration is configured in Aria Operations for Logs , the frequency of inventory updates can be decreased with the following steps:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config
  2. Under the following section, increase the poll-seconds value to 3600 or higher
<vcops>
<enabled value="true" />
....
<poll-seconds value="900" />
....
</vcops>
Notes: This value determines the frequency of sending DNS requests to build the inventory mapping between objects in Aria Operations, and logs ingested by Aria Operations for Logs .
 
This will not always be effective, and we are exploring other options for mitigating the issue when it occurs due to integration with Aria Operations
  1. SSH to the primary node as root
Note: If you do not know your root password, refer to: How to reset the root password in Aria Operations for Logs (53649)
  1. Restart the service by running the following command
service loginsight restart
  1. Repeat steps 3-4 sequentially on all nodes in the Aria Operations for Logs cluster.

How to disable:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/admin
  2. Under Integration, click on Aria Operations
  3. Click the x in the top right corner to remove the integration

How to adjust in vRealize Log Insight  8.8.2:

As of vRealize Log Insight 8.8.2, there is a new property in the <vcops> section of internal/config. It's called "matching-policy". It's used to control DNS queries done while trying to match Aria Operations resource with the source log event.

There are 3 possible values:

  • legacy
  • smart
  • off

The default value is "smart". With this option, matching will be done based on resource type. The number of queries would be 2-3 times smaller than with the "legacy" option, but the accuracy might be reduced.
"Legacy" is the same method which was in used until vRealize Log Insight 8.8.2. It does a lot of DNS queries and has the highest accuracy.
"Off" completely disables DNS queries, but not inventory matching. This has the lowest accuracy.

Which option to choose highly depends on the environment. These options offer a tradeoff between accuracy and number of DNS queries. As such, the value should be chosen based on priorities. If DNS queries are flooding your network, switch it to "off". If high number of DNS queries are fine, but the accuracy of matching is lacking, switch it to "legacy".

To change this setting, follow the steps below:

  1. Navigate to https://vRealizeLogInsightIPorFQDN/internal/config
  2. Locate the section under the <vcops> tag
  3. Modify the value for matching-policy as you see fit
  4. Click Save

Additional Information

If the above doesn't resolve the issue, please follow the below steps:
 
1.) Login to Aria Logs UI and navigate to Log Management -> Log Forwarding and under Host section, use the IP instead of the FQDN of the server 
2.) Also under the <vcops> section in the internal config page of Aria Logs, add the below line if not already present:
 
<matching-policy value="off" />
 
Impact/Risks:
These workarounds will affect how vRealize Log Insight resolves host names, including hostname of the syslog messages, Launch in Context, and Alert Integration features of the vRealize Operations integration.
Powered by Wolken Software