No active nodes found. Limit specified for max request will be applied to each node instead.

Products

• CA API Gateway

Issue/Introduction

When using the  rate limit assertion the following error is seen in the logs :

11000 Policy processing caught an exception: Unexpected error while accessing provider. No active nodes found. Limit specified for max request will be applied to each node instead.

 

 

Environment

All supported versions of the API Gateway

Cause

Per our documentation: https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-1/policy-assertions/assertion-palette/service-availability-assertions/apply-rate-limit-assertion.html 

"If you have a cluster of Gateways, the limits entered in this assertion are divided among the number of "up" nodes in the cluster. A node is considered “up” if it has posted its status within the past 8 seconds (configurable via the ratelimit.clusterStatusInterval cluster property)."

By default, the cluster info provider is 'ssgdb'. This can be found in /opt/SecureSpan/Gateway/node/default/etc/conf/system.properties file: com.l7tech.server.extension.sharedClusterInfoProvider=ssgdb

Then the status timestamp of nodes can be queried on ssg database: select name, FROM_UNIXTIME(statustimestamp/1000) statustime from cluster_info;

And compared to current time: select now();

If the time difference is greater than 8 seconds, the nodes are considered as non-active nodes. For example:

If no active node found, it will cause the error message.

Resolution

Increase the value of cluster-wide property "ratelimit.clusterStatusInterval" as per the time difference. For the above example, double the value to 16000.

The gateway nodes, by default, should update its node status every 4 seconds. The update latency could be due to a slow MySQL. If it's confirmed that the MySQL performance is poor, another solution is to fix the MySQL performance issue.