Webhook keeps failing with "read timed out" from Rally

Products

• CA Agile Central SaaS (Rally)

Issue/Introduction

Webhook keeps failing with "read timed out" from Rally 

*Note this was using  *amazonaws.com/prod/webhook

Cause

As I checked the lambda functions invocations and test the response of the lambda function on my local machine, I found that it did respond but the response time was around 2 seconds. 

Environment

Release : SAAS

Component : AGILE CENTRAL

Resolution

The lambda function invocation takes more than 2 seconds to proceed before we send a response back to the webhook call. And this delay of response will result in a "read timed out" of the webhook on the Rally side. So we’ve added a message queue for queuing all coming webhook calls and responding to the webhook right away. All messages in the queue will be handled by other lambda functions.