Logs not ingesting between clusters using Ingestion API over port 9000

Products

• VMware Aria Suite

Issue/Introduction

When using log forwarding to another Aria Operations for Logs cluster using Ingestion API (CFAPI) over port 9000, no logs are being seen on the receiving cluster end.

• You validate DNS, basic networking and check communication with telnet/curl commands using port 9000 and all seems to succeed.
• The test connection succeeds and you can see the log entry for this test on the receiving cluster
• If you switch the protocol to syslog the logs get forwarded
• You may see some events dropped in the Management > System Monitor > Statistics page

Environment

Aria Operations for Logs 8.x

Cause

• There is some blocking of port 9000 between the clusters or one or both of the clusters have been set to 'Enforce SSL-Only Connections'
  • NOTE: The option "Require SSL Connection" is enabled by default.
  • Reference: VMware Aria Operations for Logs Server Rejects the Connection for Non-Encrypted Traffic
• A certificate has not been accepted into Aria Operations for Logs when using SSL
• NSX environments can also cause some blocking, check KB - Log Insight load balancer incompatible with NSX Distributed Firewall Protection

Resolution

When configuring the log forwarding, check the box for 'Use SSL' and use default port 9543. 

 

Check Enforce SSL-Only Connections as well on both clusters.

If there is a certificate present that has not been trusted you will receive a popup message. Trust the certificate and save the configuration.

If you have a need to send non-encrypted traffic, adjust the setting for "Require SSL Connection" as mentioned in VMware Aria Operations for Logs Server Rejects the Connection for Non-Encrypted Traffic

Additional Information

Add a VMware Aria Operations for Logs Log Forwarding Destination