Best practices for External Service Metrics via RabbitMq

Products

• CA API Gateway

Issue/Introduction

We are trying to monitor service metrics via Grafana on real time. To do this, we publish service metrics to rabbitmq queue, and we collect this metrics on clickhouse and visualize on Grafana. We have nearly 800-1200 service call per second. 

• First, we tried using rabbitmq HTTP API for publishing metrics. However, the rabbitmq CPU load increased so much.
• After that, we use AMQP tactical assertion for publishing service metrics. This time rabbitmq has a good performance but the gateway load average increased 4-5 times.

Is there a fine-tuning options for route via AMQP assertion. Alternatively, can you suggest a proper and effective way to publish service metrics to external sources?

Environment

Release : 10.1, 11.x

Resolution

Performing additional routing upon every request is certainly going to add load. I would look at using the accumulate assertion:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/11-1/policy-assertions/assertion-palette/logging-auditing-and-alerts-assertions/accumulate-data-in-memory-assertion.html

Of course, this means that the graphs will not be truly real-time, the buffer holds the data, but if you extract frequently enough, it can be near real-time. For example, set the max buffer size to the size of 9 entries, when it appends the 10th, then it extracts the data, and you can route all 10 of them at once. In addition, if the buffer has 9 of them, then no one hits the policies for several minutes, it may not actually send the data until they do.