Alerts Not Generated and Notifications Not Sent in VMware Aria Operations 8.17.1 Due to Webhook Errors 'Invalid JSON Payload'

Products

• VMware Aria Operations 8.x
• VMware Aria Suite

Issue/Introduction

In VMware Aria Operations 8.17.1, users may experience an issue where alerts are no longer generated, and notifications are not being sent to the configured alerting tools. This problem is caused by a broken connection between Aria Operations and the alerting tool, as reflected by errors in the analytics logs.

The logs show the following error found in /storage/log/vcops/logs/analytics-xxxx.xxxx.xxxx.xxxx.log.

2024-07-02T10:25:57,635+0000 ERROR [pool-4854-thread-1] com.vmware.vcops.platform.notification.webhook.WebhookSender.call - Failed to send POST; status code: 404
2024-07-02T10:25:57,636+0000 ERROR [ServerConnection on port 10000 Thread 803] com.vmware.vcops.alertplugins.webhook.WebhookPlugin.test - Expected POST HTTP status code 2xx but received 404
Having \ in Notification Template leads to an Error in the Analytics log
2024-06-07T11:42:56,216+0000 ERROR [pool-29-thread-3] com.vmware.vcops.alertplugins.webhook.WebhookPlugin.createWebhookMessage - Invalid JSON payload:
{"startDate":"Fri Jun 07 07:40:27 UTC 2024","updateDate":"Fri Jun 07 07:40:27 UTC 2024","cancelDate":"Fri Jun 07 11:39:56 UTC 2024","resourceId":xxxxx","adapterKind":"VMWARE","resourceKind":"VirtualMachine","resourceName":"artavazdg__VM_OEL7-pivotal-tomcat-activeMq","Health":"1.0","Risk":"1.0","Efficiency":"1.0","impact":"health","criticality":"info","alertName":"__TEST_ALERT","type":"Application","subType":"Performance","alertId":"xxxx","status":"Canceled","recommendations":"a | [11838] <div>__1. Assign correct FQDNs to all Manager nodes and verify the DNS configuration is correct for successful lookup of all Manager nodes' FQDNs. 2. Alternatively, deactivate the use of FQDNs by invoking the NSX API PUT /api/v1/configs/management with publish_fqdns set to false in the request body.\\\ [11839] After that calls from Transport nodes and from Federation to Manager nodes in this cluster will use only IP addresses.</div>","alertURL":"https://xx.xx.xx.xxx/vcf-operations/ui/inventory;mode=hierarchy;resourceId=xxxx;tab=alerts-and-symptoms;subTab=alerts;entityId=xxxxxxx"}

Environment

Aria Operations 8.17x

Cause

The presence of special characters, such as backslashes (\), in the notification template payload. These characters lead to the generation of invalid JSON payloads, causing errors during the webhook POST requests. As a result, the webhook plugin fails with a 404 status code, and no alerts or notifications are generated.

Resolution

Upgrade to VMware Aria Operations 8.17.2

The issue has been fixed in version 8.17.2. Upgrading to this or a later version will permanently resolve the issue and prevent similar problems related to special characters in the future.

 

Additionally:

Manually removing any special characters from the notification template payload. This can temporarily resolve the issue and allow the webhook to function correctly.