RabbitMQ Cluster Outage: enospc Error on ra_log_wal Initialization

Products

• VMware Tanzu RabbitMQ

Issue/Introduction

• RabbitMQ node (e.g., server-0) fails to start.
  
  
• Error logs show a badmatch error during the initialization of the ra_log_wal (Write-Ahead Log).
  
  
• Specific error string: {{error,{enospc, ...}}, ...}.
  
  
• Monitoring indicates the disk partition (typically /var/lib/rabbitmq) is at 100% utilization.

Environment

VMware Tanzu RabbitMQ K8S

Cause

The RabbitMQ node has run out of disk space. This is often caused by

1. Quorum Queue Growth: High message volume or slow consumers causing Quorum Queue segment files to accumulate.
   
   
2. Insufficient Provisioning: The initial disk allocation was exceeded by the persistent message store and log files.
   
   
3. Log Accumulation: Excessive logging or lack of log rotation filling the root or data partition.

Resolution

1. Immediate Recovery (Disk Expansion)

If the environment supports dynamic volume expansion (e.g., Kubernetes with PVCs):

1. Expand PVC: Update the Persistent Volume Claim (PVC) size for the affected StatefulSet.
   
   
2. Rolling Restart: Perform a rolling restart of the StatefulSet pods to trigger the underlying filesystem expansion.
   bash

    

   kubectl rollout restart statefulset <rabbitmq-cluster-name>

    

3. Verify Expansion: Once the pods are back in a Running state, verify the new disk size:
   bash

    

   kubectl exec <pod-name> -- df -h /var/lib/rabbitmq

1. rabbitmqctl force_boot

Prevention

• Monitor Disk Alarms: Ensure disk_free_limit is configured and monitored to block publishers before the disk is completely full.
  
  
• Over-provisioning: Always over-provision disk space when using Quorum Queues, as they require significant overhead for WAL and segment files.