RabbitMQ: Troubleshooting topic exchanges

Products

• RabbitMQ

Issue/Introduction

RabbitMQ clusters that use topic exchanges can sometimes see slowness during binding operations. RabbitMQ uses a Trie structure to manage topic routing keys. Complex routing keys can cause routing and binding to be computationally intensive.

In this article, we will go over commands that can evaluate the complexity of these keys and, in turn, identify the cause of slowness.

Environment

All supported RabbitMQ versions

Resolution

 

1. Capture Mnesia Table Sizes

This command captures the sizes of key topic exchange routing-related Mnesia tables. If you see a high number of rabbit_topic_trie_node entries compared to your total bindings, it usually suggests your routing keys have a very high "cardinality", which prevents the Trie from being efficient.

rabbitmqctl eval '[{T, mnesia:table_info(T, size)} || T <- [rabbit_topic_trie_node, rabbit_topic_trie_edge, rabbit_topic_trie_binding]].'

2. Check for Lock Contention and Dumper Activity

If you suspect the node is hanging or slow to process binding updates, check for lock contention or high "dumper activity". These commands are read-only and should be run during the operations being compared (e.g., during binding population).

• To identify if Mnesia is waiting on resource locks:

  rabbitmqctl eval 'mnesia:system_info(held_locks).'
  

• To monitor the "dumper" (moving data from write-ahead logs to disk):

  rabbitmqctl eval 'mnesia_dumper:get_log_writes().'
  

3. Evaluate Wildcard Complexity

The # pattern is significantly more expensive than * pattern because it forces the router to explore more branches of the Trie.

• List keys and associated binding counts for # patterns:

  rabbitmqctl eval '[{K, length(Bs)} || #binding{key = K} = B <- ets:tab2list(rabbit_route), Bs <- [[B]], binary:match(K, <<"#">>) =/= nomatch].'
  

• Get a total count of bindings containing #:

  rabbitmqctl eval 'length([1 || #binding{key = K} <- ets:tab2list(rabbit_route), binary:match(K, <<"#">>) =/= nomatch]).'
  

4. System-level Monitoring

If the commands above show high dumper activity, verify the physical disk impact using standard OS tools while performing operations like mass binding creation or high-throughput routing.

• Linux (iostat):

  iostat -x 1 /path/to/node/data/dir