Log Forwarding to Splunk Fails or Hangs During Test Connection

Products

• VCF Operations

Issue/Introduction

When attempting to configure a new Log Forwarding destination to a Splunk server in VMware Aria Operations for Logs, the test connection may fail or hang indefinitely. This prevents the successful delivery of logs to the remote Syslog/Splunk ingestion point.

Symptoms

• Navigating to Log Management > Log Forwarding > New Destination and entering Splunk details results in a "Test Connection" failure.

• After confirming the port is open, the "Test Connection" button results in a continuous loading spinner and does not time out.

• Logs are not received by the Splunk indexer.

Environment

Aria Operations for Logs 8.x

Cause

These symptoms are generally caused by a breakdown in the SSL/TLS Handshake or a Port blocking issue or an underlying DNS resolution conflict. Common root causes include:

1. Incorrect DNS Records: The destination FQDN resolves to multiple IP addresses, or the Reverse Lookup (PTR) does not match the Forward Lookup (A-record), causing the source to reject the identity of the destination.

2. Incomplete TLS Handshake: A firewall or network security appliance could be dropping packets from/to the destination.

3. Certificate Mismatch: The destination server is not configured to present a certificate that matches the FQDN/IP specified in the Aria Operations for Logs configuration.

Resolution

1. Validate Network Layer Connectivity

Ensure that the destination port is reachable from all nodes in the Aria Operations for Logs cluster.

• Run the following commands via SSH:
  
  nc -zv <Destination_FQDN> <Port>

curl -v telnet://<FQDN>:443

2. Verify DNS and PTR Consistency

Verify that the source and destination can uniquely identify each other.

• Run nslookup <Destination_FQDN> to ensure it returns a single, expected IP.

• Run nslookup <Destination_IP> to ensure the PTR record matches the FQDN exactly.

3. Debug the SSL Handshake

Use the OpenSSL client to determine where the handshake is failing:

openssl s_client -connect <Destination_FQDN>:<Port> -debug

• If it hangs at "SENT Client Hello": An intermediate firewall is likely dropping the return traffic from the destination.

• If it returns "No Certificate Found": The destination server's listener is not correctly configured with a valid SSL certificate.

4. Perform Packet Capture Analysis

If the handshake hangs, capture the traffic to identify where the "Server Hello Done" message is lost:

1. Start a capture on the Aria appliance: tcpdump -i eth0 host <Destination_IP> and port <Port> -w /tmp/forwarding_issue.pcap

2. Attempt the "Test Connection" in the UI.

3. Stop the capture and analyze.

Additional Information

How to Install TCPDUMP package onto Aria Operations