How to Generate CPU Flame Graphs for RabbitMQ Performance Analysis using Linux Perf

Products

• VMware Tanzu RabbitMQ

Issue/Introduction

When troubleshooting performance bottlenecks or high CPU utilization in RabbitMQ, standard system monitoring tools (like top or htop) only show that the beam.smp process is consuming resources. They do not reveal which specific internal Erlang functions or operations are causing the load.

This article provides a comprehensive guide on how to configure RabbitMQ for profiling, run a stress test using perf-test, and generate CPU Flame Graphs. Flame Graphs provide a visual, interactive call-stack representation, enabling deep-dive analysis into RabbitMQ's CPU consumption.

Environment

NA

Cause

High CPU usage in RabbitMQ can be caused by various factors, including complex routing topologies, excessive garbage collection, high connection churn, or storage bottlenecks. Capturing a Flame Graph is a diagnostic step required to pinpoint the exact code path or Erlang process responsible for the bottleneck so that appropriate architectural or configuration changes can be made.

Resolution

Steps:

1. Since we are going to stress test RabbitMQ and we do not want RabbitMQ to artificially slow down our performance tests by protecting itself against overload, we increase the memory threshold to not hit memory alarms, and increase credit flow control settings by a factor of 4 compared to their default settings. You can also try out setting credit_flow_default_credit to {0, 0} which disables credit based flow control altogether. Create the following advanced.config file:

[
 {rabbit,[
  {vm_memory_high_watermark, {absolute, 15_000_000_000}},
  {credit_flow_default_credit, {1600, 800}}
 ]}
].
 
2. We set Erlang emulator flags +JPperf true to enable support for Linux perf and +S 4 to create 4 scheduler threads. You can configure it in rabbitmq-env.conf using below:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+JPperf true +S 4"

3. Create a RabbitMQ cluster as per your usual steps, sample command:
podman run --network perf-test --pid=host --name rabbitmq -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+zdbbl 512000 +JPperf true +S 4" -p 15672:15672 -p 5672:5672 -v /opt/rabbitmq/advanced.config:/etc/rabbitmq/advanced.config:ro rabbitmq.packages.broadcom.com/vmware-tanzu-rabbitmq:4.1.4

--network perf-test ==> Attach the container to an existing user-defined network named perf-test so it can communicate with other containers/services on that network by name.
--pid=host ==> Share the host’s PID namespace; processes inside the container see host PIDs. Useful for host-level monitoring or debugging, but weaker isolation.
​--name rabbitmq ==> Give the container the name rabbitmq so you can refer to it with podman stop rabbitmq, podman logs rabbitmq, etc.
-p 15672:15672 ==> Map host port 15672 to container port 15672, exposing the RabbitMQ Management UI on http://<host>:15672.
-p 5672:5672 ==> Map host port 5672 to container port 5672, exposing the AMQP port to applications on the host/network.
-v /opt/rabbitmq/advanced.config:/etc/rabbitmq/advanced.config:ro ==> Bind-mount a host file into the container.

4. Added a User and give enough privileges:
podman exec -it rabbitmq bash
env | grep ARGS
podman inspect rabbitmq

rabbitmqctl add_user user1 password1
rabbitmqctl set_permissions -p / user1 ".*" ".*" ".*"
rabbitmqctl set_user_tags user1 management

5. Install perf tool using below:
sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`

6. Install perf-test
podman run image docker.io/pivotalrabbitmq/perf-test:latest
chmod +x perf-test

7. run the perf test, sample command:
podman run --rm --network network-perftest --name perf-test1 docker.io/pivotalrabbitmq/perf-test:latest --uri amqp://user1:password1@10.89.1.2:5672 \
  --queue my-quorum \
  --variable-size 1000:30 \
  --queue-args x-queue-type=quorum \
  --auto-delete false \
  --flag persistent \
  --producers 1 \
  --confirm 2000 \
  --consumers 0 \
  --time 60

8. While PerfTest is running, in a 3rd shell window, record a profile which samples CPU stack traces at 999 hertz (--freq) recording call-graph for both kernel space and user space (-g) for 30 seconds.
sudo perf record --pid $(pgrep beam.smp) --freq 999 -g -- sleep 60

Get the pid using the below:
ps -ef | grep 'beam.smp'

You can also 
podman exec -it rabbitmq bash
cat /var/lib/rabbitmq/mnesia/rabbit@$(hostname --short).pid

9. Once the 60 seconds PerfTest run finishes, check the results. The previous perf command outputs a file perf.data. 

10. Get FlameGraph
git clone https://github.com/brendangregg/FlameGraph.git

11. Convert perf.data (created by perf record) to trace output
sudo perf script > out.perf

12. Collapse multiline stacks into single lines
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded

13. Merge scheduler profile data
sed -e 's/^[0-9]\+_//' -e 's/^erts_\([^_]\+\)_[0-9]\+/erts_\1/' out.folded > out.folded_sched

14. Create the SVG file
./FlameGraph/flamegraph.pl --title="CPU Flame Graph" out.folded_sched > cpu.svg

15. Opening the resulting cpu.svg file in your browser or you can this file with RabbitMQ Support team.