Best practices for External Service Metrics via RabbitMq

Products

• CA API Gateway

Issue/Introduction

We are trying to monitor service metrics via grafana on real time. To do this, we publish service metrics to rabbitmq queue, and we collect this metrics on clickhouse and visualize on grafana. We have nearly 800-1200 service call per second. 

First we try to use rabbitmq HTTP API for publishing metrics. but the rabbitmq CPU load increased so much.

After that we use AMQP tactical assertion for publishing service metrics. This time  rabbitmq has  a good performance but layer7 gateways load average increased 4-5 times.

İs there a fine tuning options for route via AMQP assertion. Or can you suggest a proper and effective way to publish service metrics to external sources        

Environment

Release : 10.1

Resolution

Performing additional routing upon every request is certainly going to add load.I would look at using the accumulate assertion

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-1/policy-assertions/assertion-palette/logging-auditing-and-alerts-assertions/accumulate-data-in-memory-assertion.html

Of course this means that the graphs will not be truly realtime, since its being buffered, but if you extract frequently enough, it can be near realtime.For example, set the max buffer size to the size of 9 entries, when it appends the 10th, then it will be extracted, and you can route all 10 of them at once. Also, if its buffered 9 of them, then no one hits the policies for several minutes, it may not actually send the data until they do.