Occasional Routing Failures due to connections to incorrect ports
Article ID: 104137
Updated On:
Products
CA API Gateway (Layer 7)
SA94 to API SECURITY
STARTER PACK-7
CA Rapid App Security
MOBILE API GATEWAY
CA Mobile - API Gateway
CA API Gateway
Issue/Introduction
We have a policy which accounts for 99% of our traffic, and will route requests to one of 3 hosts
Host 1 on port 8081 (just over 50%)
Host 2 on port 80 (just under 50%)
Host 3 on port 8080 (about 1%)
We are seeing low rates (1-3 per hour per node) of routing errors for host 2 where we get the following log:
2018-01-24T12:36:16.295-0800 WARNING 1443 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to http://host2/path. Error msg: Unable to obtain HTTP response from http://host2/path: Connect to host:8081 timed out
We find that occasionally Route to backend host2 will use the wrong port #. (using port : 8080 or port: 8081 rather than :80).
Cause
Reproducing this locally we have setup and do see
This happens with we have three different backend hosts and on two of them we have port specified, but on the third we do not specify a port.
In the example below we have, possible destination hosts of :
desthost=http://backend.example.com
desthost=http://localhost:8081
desthost=http://localhost:8080
We run jmeter script to do load (customer has his daily load).
Then occasionally, 16 times in 100,000 total requests (or in 33,333 requests to each backend) we get errors and find that Gateway tried to connect to :
http://odoma04-vm2.ca.com:8081
or:
http://odoma04-vm2.ca.com:8080
This happens with we have three different backend hosts and on two of them we have port specified, but on the third we do not specify a port.
In the example below we have, possible destination hosts of :
desthost=http://backend.example.com
desthost=http://localhost:8081
desthost=http://localhost:8080
We run jmeter script to do load (customer has his daily load).
Then occasionally, 16 times in 100,000 total requests (or in 33,333 requests to each backend) we get errors and find that Gateway tried to connect to :
http://odoma04-vm2.ca.com:8081
or:
http://odoma04-vm2.ca.com:8080
Environment
APIM Gateway 9.2
APIM Gateway 9.3
APIM Gateway 9.3
Resolution
Workaround :
Changing :
desthost=http://backend.example.com
desthost=http://localhost:8081
desthost=http://localhost:8080
to:
desthost=http://backend.example.com:80
desthost=http://localhost:8081
desthost=http://localhost:8080
This was tested and resolved the issue.
desthost=http://backend.example.com
desthost=http://localhost:8081
desthost=http://localhost:8080
to:
desthost=http://backend.example.com:80
desthost=http://localhost:8081
desthost=http://localhost:8080
This was tested and resolved the issue.
Resolution:
Engineering recomended the following change :
Add :
com.l7tech.common.http.prov.apache.CommonsHttpClient.enableTrace=false".
to :
/opt/SecureSpan/Gateway/node/default/etc/conf/system.properties
And restart the gateway service.
This was also tested and resolved the issue.
Add :
com.l7tech.common.http.prov.apache.CommonsHttpClient.enableTrace=false".
to :
/opt/SecureSpan/Gateway/node/default/etc/conf/system.properties
And restart the gateway service.
This was also tested and resolved the issue.
A fix will also be applied in some future version of the gateway.
Feedback
Powered by
