AWS Solution Kit multiple Lambda initiations from a single request, duplicated Lambda calls.

Products

• CA API Gateway

Issue/Introduction

Description: * 

An undocumented default behavior of the AWS Solution Kit v1.0.02 is to retry a long running synchronous Lambda after  default 50 seconds if no response has been received.  The Gateway will continue to retry every 50s until it reaches the default 200s read timeout or one of them responds without logging that it was sending a new request. 

One call to to the assertion to initiate a Lambda may spawn up to 4 executions (at 0, 50, 100, and 150 seconds) that will run to completion (presumably), but only a read timeout or a response from one of the calls will stop the assertion from sending another request.

This has cost user dollars due to duplicate calls to a 3rd party with charge per call.

A quick google search suggests that this behavior may be typical of the underlying AWS SDK, however it is not what a gateway user would expect and coupled with the complete lack of logging of the retries made it virtually impossible to track down.  So for years it was believed to be an AWS issue rather than a problem with the gateway and work arounds were implemented.

 

Environment

Gateway 10.x, 11.x with AWS solution kit v1.0.x 

Cause

AWS Solution Kit will do retry for a long running synchronous Lambda after default setting seconds if no response has been received. 

Resolution

It was recently determined that the socket timeout (property aws.socketTimeoutMillis defaults to 50s and when it expires nothing is logged and the the assertion fires off another request to AWS.  This repeats until the request timeout (aws.requestTimeoutMillis) default of 200s is reached.

However explicitly setting the gateway cluster wide property aws.socketTimeoutMillis to any value, higher or lower, will cause the behavior to change; instead it will return an error once the timeout is reached and no additional Lambda calls are made.  So reset the aws.socketTimeoutMillis value can provide a work around for the duplicated Lambda calls. 

Additional Information

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-management-aws-solution-kit/1-0/consume-aws-services/aws-assertion.html