Need helps to review data ingestion between SaaS and On-Prem Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

SaaS is configured to forward log to OnPrem Aria Operations for Logs via Log Agents.  However, the ingested data is showing a delay and not real time result.

Environment

Aria Operations for Logs 8.x

Aria Operations for Logs SaaS

Cause

High volume of data ingestion from the SaaS to the OnPrem environment.  

The endpoint configuration has issues and was throttling back the log-agent to have timeouts when forwarding the logs to destination endpoint. This cause a huge back pressure and created backlog. So when log-agent tying to connect back and forward data it was backfilling old data and hence Customer sees the delay.

Also the log-agent by default using different/standard sessions to connect to the destination, but with this recent upgrade using the websocket can handle 10X volume of data forward at much faster rate

Resolution

The following workaround can upscale the performance on the connectivity (sessions) to improve the log forwarding and resolve the lag and delay issue:

1. Login to the Virtual machine where the corresponding log-agent is deployed.
2. Navigate/change directory to the path /etc/systemd/system/ where the systemd unit file for the log-agent docker service is available.
   1. Command: cd /etc/systemd/system/.
3. Edit the log-agent.service file.
4. Add the below environment variable to the docker command -e ENABLE_WEB_SOCKET_SESSIONS=true under the ExecStart= section. Make sure a space before and after this command.
5. Save the changes to this systemd unit file.
6. Once save successfully, restart the log-agent.service.
   1. Command: systemctl daemon-reload.
   2. Followed by Command: systemctl restart log-agent.service.
7. This will restart the log-agent docker service with the updated environment variable settings.
8. Check if the log-agent docker is running and active via command "docker ps". Also can try running the command "systemctl is-active log-agent.service". This should return active status.
9. Verify in the UI as well under https://www.mgmt.cloud.vmware.com/li/vmc-log-forwarder/vmc, whether the restarted log-agent is active.
10. Also, check the latest log-agent logs to see if the message WSMessageConsumer and WebSocket connection is established successfully are logged. Command: tail -f /data1/log-agent/log-forwarder.log.
11. Please repeat this process for all the log-agents deployed.

Screenshot showing the modification to the log-agent.service file:

Validate and confirm log ingestion are showing Real Time data in the OnPrem environment.