Aria Automation/Orchestrator Slowness and High CPU Due to fluentd crash

Products

• VCF Automation

Issue/Introduction

In VMware Aria Automation and VMware Aria Automation Orchestrator, enabling the Aria Operations for Logs integration results in sustained high CPU usage across all cluster nodes and general system slowness.

• Sluggish vracli outputs.

• Aria Automation Orchestrator workflows running slowly.

• The fluentd log forwarder enters a continuous crash loop.

Reviewing the logs reveals that the /services-logs/logging/fluentd.log has the below error snippet

error_class=NoMethodError error="undefined method `escape' for URI:Module"
/etc/fluent/plugin/out_loginsight_http.rb:143:in `format_url'
/etc/fluent/plugin/out_loginsight_http.rb:117:in `start'

Environment

 

VMware Aria Automation 8.18.1 Patch 5

VMware Aria Automation Orchestrator 8.18.1 Patch 6

 

Cause

The fluentd log forwarder crash loop is caused by an incompatibility in the out_loginsight_http.rb plugin.

Resolution

The issue will be resolved the next release of Aria automation/Orchestrator

Implement one of the following two workarounds:

Method 1: Disable Aria operations for logs Integration

1. Log into the Aria Automation node via SSH using root credentials.

2. Execute the following command to remove the integration configuration:

   vracli vrli unset
   

Method 2: Manually Patch the Plugin File

Note: Before applying any changes, please ensure you take a backup of the configuration file and a snapshot of all nodes (without memory)

1. Log into the affected node(s) via SSH as root.

2. Open the file /etc/fluent/plugin/out_loginsight_http.rb using a text editor such as vi.

3. Locate the following code string:

   URI.escape(agent_id)
   

4. Manually modify the string to use the default parser instead:

   URI::DEFAULT_PARSER.escape(agent_id)
   

5. Save and close the file.