Webhook Notification Plugin fails to send all notifications

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Webhook configured to send notification to Splunk servers.

• Initial test connection failed with

  "[Failed to send Webhook: java.net.SocketTimeoutException: Read timed out]" 

• Intermittent connection failures causing some notifications not being received at Splunk endpoint.
• Email notifications for an alert seem to be working fine but the webhook notification is not sent to the endpoint

Environment

Aria Operations 8.x

Cause

• TCP Keepalive issue on the Cloud Proxy since the Cloud Proxy is selected for the Webhook Notification.
• Sometimes if there is a misconfiguration of an alert notification, it can cause things to get stuck in memory. In such cases, a simple restart of the cluster can resolve the issue. See Rebooting nodes in Aria Operations

Resolution

Lower the standard keepalive from 7200 to 1800 ms on the Cloud Proxy.

• Command to set the keep alive on the Cloud Proxy console:
  

  sysctl -w net.ipv4.tcp_keepalive_time=1800

• Command to review change that has been made:
  

  cat /proc/sys/net/ipv4/tcp_keepalive_time

• To make the change persist after reboot, update the "/etc/sysctl.conf" file.
  • Add the following line to the end of the "/etc/sysctl.conf" file.
    

    net.ipv4.tcp_keepalive_time=1800

Additional Information

If the steps in this KB do not help resolve your webhook notification issue, try Rebooting nodes in Aria Operations