Error "Test was not successful: Failed to post to the server." occurs while trying to integrate Splunk in Aria Operations for Outbound settings

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Users are unable to successfully integrate Splunk via the Webhook plugin in Outbound settings using Bearer token authentication.
• When attempting to test the connection, the UI throws the following error: Test was not successful: Failed to post to the server.
• Test Connection fails but using the same Outbound setting for the notification rule succeeds and the notification is created successfully.
• Additionally, errors such as ConnectTimeoutException or HTTP 401/403 rejections may be observed in the /storage/log/vcops/log/vcops-bridge.log files:
  

  YYYY-MM-DDT00:25:33,223+0000 ERROR [ServerConnection on port 10000 Thread 5] com.vmware.vcops.bridge.server.BridgeTracerAspect.processBridgeResult - Failed to post to the server.[Failed to send Webhook: org.apache.http.conn.ConnectTimeoutException: Connect to <Splunk-FQDN> [<Splunk-FQDN>/##.##.##.##, <Splunk-FQDN>/##.##.##.##] failed: connect timed out]
  YYYY-MM-DDT00:25:33,223+0000 ERROR [ServerConnection on port 10000 Thread 5] com.vmware.vcops.bridge.server.BridgeTracerAspect.processBridgeResult - SessionId: ##.##.##.##::##.##.##.##; UserId: ##.##.##.##; UserName: admin

Environment

Aria Operations 8.18.x

Cause

This issue occurs due to a combination of payload formatting, authentication mismatches, and incorrect endpoint configurations:

1. Payload Format: Splunk's HTTP Event Collector (HEC) enforces a strict payload format, requiring all event data to be wrapped inside a top-level "event" key. The default templates send the alert payload unwrapped, which Splunk's HEC automatically rejects.

2. Authentication Scheme: Configuring a "Bearer Token" in the Service Credentials hardcodes a Bearer <token> header. Splunk HEC strictly requires the Splunk <HEC-token> format and will reject standard bearer prefixes with a 401/403 error.

3. URL Port: Including explicit/custom ports (such as 7777) for the Splunk Cloud endpoint causes connection timeouts.

Resolution

To resolve this issue, you must configure a custom payload template and adjust your credential and URL settings to meet Splunk's HEC requirements.

Step 1: Create an API Key Service Credential Do not use the "Bearer Token" credential type. Instead, create a Service Credential of type API Key:

• Key: Authorization

• Value: "Splunk <your-HEC-token>" (This sends the key/value pair as-is, ensuring the correct scheme required by Splunk).

Step 2: Create a Custom Payload Template Replace the default template with a Splunk-compatible one. Wrap the placeholder data inside the "event" key:

JSON file:

{
  "event": {
     "alertId": "${ALERT_ID}",
     // any other placeholder
  }
}

Note: Additional placeholders can be added freely as long as they remain inside the "event" object.

Step 3: Update the HEC URL Use the dedicated HEC endpoint without any custom port (do not use the base Splunk Cloud stack URL).

• Correct Format: https://http-inputs-<stack>.splunkcloud.com/services/collector/event

Step 4: Create the Notification Create a new notification, attach it to the Splunk plugin instance, and select your newly created Splunk-compatible webhook payload template. Proceed to test the notification and it should succeed.

Additional Information

Plugin test would fail in any case (not only while using Bearer token), because Splunk is expecting to get a template which has different format from the one that Aria Operations is sending.