Alerts not generated and notifications not sent in VMware Aria Operations 8.17.1 due to webhook errors 'Invalid JSON Payload'

Products

• VMware Aria Suite

Issue/Introduction

In VMware Aria Operations 8.17.1 or older versions, users may experience an issue where alerts are no longer generated, and notifications are not being sent to the configured alerting tools. This problem is caused by a broken connection between Aria Operations and the alerting tool, as reflected by errors in the analytics logs.

 

The logs show the following error found in /storage/log/vcops/logs/analytics-xxxx.xxxx.xxxx.xxxx.log:

ERROR [pool-xxxx-thread-x] com.vmware.vcops.platform.notification.webhook.WebhookSender.call - Failed to send POST; status code: 404
ERROR [ServerConnection on port 10000 Thread xxx] com.vmware.vcops.alertplugins.webhook.WebhookPlugin.test - Expected POST HTTP status code 200 but received 404

 

Having a backslash symbol "\" in notification template leads to an error in the analytics log as below:

ERROR [pool-xx-thread-x] com.vmware.vcops.alertplugins.webhook.WebhookPlugin.createWebhookMessage - Invalid JSON payload:
{"startDate":"xxx xxx xx xx:xx:xx UTC xxxx","updateDate":"xxx xxx xx xx:xx:xx UTC xxxx","cancelDate":"xxx xxx xx xx:xx:xx UTC xxxx","resourceId":xxxxx","adapterKind":"VMWARE","resourceKind":"VirtualMachine","resourceName":"xxxxxx__VM_OEL7-pivotal-tomcat-activeMq","Health":"1.0","Risk":"1.0","Efficiency":"1.0","impact":"health","criticality":"info","alertName":"__TEST_ALERT","type":"Application","subType":"Performance","alertId":"xxxx","status":"Canceled","recommendations":"a | [xxxxx] <div>__
1. Assign correct FQDNs to all Manager nodes and verify the DNS configuration is correct for successful lookup of all Manager nodes' FQDNs. 
2. Alternatively, deactivate the use of FQDNs by invoking the NSX API PUT /api/v1/configs/management with publish_fqdns set to false in the request body.\\\ [xxxxx] After that calls from Transport nodes and from Federation to Manager nodes in this cluster will use only IP addresses.</div>","alertURL":"https://xxx.xxx.xxx.xxx/vcf-operations/ui/inventory;mode=hierarchy;resourceId=xxxx;tab=alerts-and-symptoms;subTab=alerts;entityId=xxxxxxx"}

Environment

Aria Operations 8.17.x and Older

Cause

The presence of special characters, such as backslashes (\), in the notification template payload. These characters lead to the generation of invalid JSON payloads, causing errors during the webhook POST requests. As a result, the webhook plugin fails with a 404 status code, and no alerts or notifications are generated.

Resolution

Upgrade to VMware Aria Operations 8.17.2

This issue is known and has been fixed in version 8.17.2. Upgrading to this or a later version will permanently resolve the issue and prevent similar problems related to special characters in the future.

Additional Information

Manually removing any special characters from the notification template payload. This can temporarily resolve the issue and allow the webhook to function correctly.