Webhook alert notification are not received randomly from Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

You have set up Notifications rule in Aria Operations to send out Alerts from Aria Operations via a webhook notification plugin, However not all expected alerts are received at the webbook endpoint.

Issue is occurring randomly at different time intervals 

Environment

Aria Operations 8.18.x

Cause

Webhook endpoint might be rejecting POST API calls from Aria Operations.

 

Resolution

To find the root cause of the issue perform below steps:

• Go to Administration ->Control Panel -> Support logs and expand the primary node logs. 
• Click on the Analytics folder and then click on the gear symbol on the top right for log settings.
• Add below class in the logs and set logging level to DEBUG.

com.vmware.vcops.platform.notification.webhook.WebhookSender

• Reproduce the issue and find an Alert in the Operations -> Alerts page which is not received at the webbook endpoint. Please take a note of the 'Created On' field and 'Alert Id'  fields
• Open support logs page again and under the analytics logs on primary node you will find a file like analytics-xxxxxxx.log , open this and go back to the time close to 'Created On' time noted earlier. Example below:
• 'You should find a log entry like below. match the fields 'VROPS_ALERT_TIME' and 'entityId' with above fields in the UI
  

  2025-04-15T13:20:06,853+0000 DEBUG [pool-29-thread-3]  com.vmware.vcops.platform.notification.webhook.WebhookSender.send - Queueing Webhook Message
  Webhook Message is: {"eventsourcesendingserver":"<Aria Ops FQDN>","eventsourceexternalid":"aa8a6d3c-d071-4615-9928-4fa67526279d","eventsourcetag":"vROPs-source","title":"siteX VM CPU utilization has reached a critical level-aa8a6d3c-d071-4615-9928-4fa67526279d","node":"vrops_node","category":"vROPs-xxxx","object":"VMWARE","application":"vROPs Webhook Plugin to Event Handler","longDescription":"VROPS_RESOURCE_ID : c7385f8a-53ca-4b82-8674-d2d69a29985b / VROPS_TYPE : Application / VROPS_SUBTYPE : Capacity /  VROPS_CRITICALITY : critical / VROPS_ALERT_TIME Tue Apr 15 13:20:04 UTC 2025 / VROPS_RESOURCE_KIND : VirtualMachine / ALARMS_INFO: Symptom Set - self\nConditions\xxxoperations\/ui\/inventory;mode=hierarchy;resourceId=c7385f8a-53ca-4b82-8674-d2d69a29985b;tab=alerts-and-symptoms;subTab=alerts;entityId=aa8a6d3c-d071-4615-9928-4fa67526279d"}
  Uri is: <webhook endpoint URL>
  Headers are: {Accept=*/*}
  Method is: POST
  Content Type is: application/json

• Just after this log you will find a debug log stating the reason for failure of POST from Aria OPs 
  

  2025-04-15T13:20:06,854+0000 DEBUG [pool-45-thread-7]  com.vmware.vcops.platform.notification.webhook.WebhookSender.debug - Sending: POST https://<webhook_endpoint_URL>
  2025-04-15T13:20:06,854+0000 ERROR [pool-45-thread-7]  com.vmware.vcops.platform.notification.webhook.WebhookSender.call - Failed to complete POST
  org.apache.http.conn.HttpHostConnectException: Connect to <webhook_endpoint hostname:port /IP] failed: Connection refused (Connection refused)

As you can see here the reason for Aria Operations not able to push the alert notification is that 'connection refused'. Please work with your webhook endpoint team to see why it is refusing these connections from Aria Operations.