Occasional Routing Failures due to connections to incorrect ports

Products

• STARTER PACK-7
• CA Rapid App Security
• CA API Gateway

Issue/Introduction

We have a policy which accounts for 99% of our traffic, and will route requests to one of 3 hosts
Host 1 on port 8081 (just over 50%)
Host 2 on port 80 (just under 50%)
Host 3 on port 8080 (about 1%)

We are seeing low rates (1-3 per hour per node) of routing errors for host 2 where we get the following log:

2018-01-24T12:36:16.295-0800 WARNING 1443 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to http://host2/path. Error msg: Unable to obtain HTTP response from http://host2/path: Connect to host:8081 timed out

We find that occasionally Route to backend host2 will use the wrong port #.  (using port : 8080 or port: 8081 rather than :80). 


 

Cause

Reproducing this locally we have setup and do see 

This happens with we have three different backend hosts and on two of them we have port specified, but on the third we do not specify a port. 

In the example below we have, possible destination hosts of  : 
   desthost=http://backend.example.com
   desthost=http://localhost:8081
   desthost=http://localhost:8080

We run jmeter script to do load (customer has his daily load).  

Then occasionally, 16 times in 100,000 total requests (or in 33,333 requests to each backend) we get errors and find that Gateway tried to connect to : 
   http://odoma04-vm2.ca.com:8081
or: 
   http://odoma04-vm2.ca.com:8080


 

Environment

APIM Gateway 9.2 
APIM Gateway 9.3

Resolution

Workaround : 

Changing : 
   desthost=http://backend.example.com
   desthost=http://localhost:8081
   desthost=http://localhost:8080

to: 
   desthost=http://backend.example.com:80
   desthost=http://localhost:8081
   desthost=http://localhost:8080

This was tested and resolved the issue. 

Resolution: 

Engineering recomended the following change : 
Add : 
   com.l7tech.common.http.prov.apache.CommonsHttpClient.enableTrace=false". 
to :  
   /opt/SecureSpan/Gateway/node/default/etc/conf/system.properties 

And restart the gateway service. 

This was also tested and resolved the issue. 

A fix will also be applied in some future version of the gateway.