Occasional routing failures due to connections to incorrect ports

Products

• CA API Gateway

Issue/Introduction

You have a policy which accounts for 99% of our traffic and will route requests to one of 3 hosts.

• Host 1 on port 8081 (just over 50%)
• Host 2 on port 80 (just under 50%)
• Host 3 on port 8080 (about 1%)

You are seeing low rates (1-3 per hour per node) of routing errors for host 2 where you get the following log:

2018-01-24T12:36:16.295-0800 WARNING 1443 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to http://host2/path. Error msg: Unable to obtain HTTP response from http://host2/path: Connect to host:8081 timed out

You find that occasionally Route to backend host2 will use the wrong port #.  (using port : 8080 or port: 8081 rather than: 80). 

Environment

APIM Gateway 10.x

Cause

Reproducing this locally we see this happens with we have three different backend hosts and on two of them we have a port specified, but on the third, we do not specify a port. 

In the example below we have, possible destination hosts of : 
   desthost=http://backend.example.com
   desthost=http://localhost:8081
   desthost=http://localhost:8080

We run jmeter script to do the load:

Then occasionally, 16 times in 100,000 total requests (or in 33,333 requests to each backend) we get errors and find that Gateway tried to connect to : 
   http://<Domain>-machine.example.com:8081
or: 
   http://<Domain>-machine.example.com::8080

Resolution

Workaround : 

Change : 
   desthost=http://backend.example.com
   desthost=http://localhost:8081
   desthost=http://localhost:8080

to: 
   desthost=http://backend.example.com:80
   desthost=http://localhost:8081
   desthost=http://localhost:8080

Resolution: 

Engineering recommends the following change : 
Add : 
   com.l7tech.common.http.prov.apache.CommonsHttpClient.enableTrace=false". 
to :  
   /opt/SecureSpan/Gateway/node/default/etc/conf/system.properties 

And restart the gateway service. 

A fix will also be applied in a version of the gateway.