RabbitMQ Connection Closed Abruptly (noproc) due to Celery Heartbeat Starvation

Products

• RabbitMQ
• VMware Tanzu RabbitMQ

Issue/Introduction

A Python/Celery client suddenly loses its RabbitMQ connection. The broker logs show a channel process crashing with exception exit: noproc while trying to clean up after receiving a shutdown from its parent connection.

Example Log Snippet:

[error] <0.XXXXXXX.0> ** Generic server <0.XXXXXXX.0> terminating
[error] <0.XXXXXXX.0> ** Reason for termination == noproc
[error] <0.XXXXXXX.0>     exception exit: noproc
[error] <0.XXXXXXX.0>       in function gen_server2:terminate/3

Environment

• Python + Celery
• OAuth2 auth backend
• Low/aggressive broker_heartbeat (e.g. 10s)
• Affects all RabbitMQ versions

Cause

Heartbeat Starvation caused by Python’s GIL. Long-running synchronous tasks (API calls, DB queries, heavy processing) block the worker thread, preventing it from sending heartbeats. RabbitMQ declares the connection dead and closes it, leading to the noproc channel crash during cleanup.

Resolution

1. Increase Heartbeat Interval (recommended: 60s)

The exact parameter name depends on the Celery version running in the client application environment:

For Celery 4.0 and Newer (Modern): Use lowercase settings.
broker_heartbeat = 60

For Celery 3.x and Older (Legacy): Use uppercase settings.
BROKER_HEARTBEAT = 60

2. Add Task Time Limits

@app.task(time_limit=30, soft_time_limit=25)
def process_data_task():
    …

3. Audit Logs Correlate RabbitMQ error timestamps with Celery worker logs to identify long-running tasks that exceeded the heartbeat window.

Additional Information

Related Documentation:

• RabbitMQ Heartbeats
• Celery Broker Settings
• Broadcom KB Article: Question about rabbitmq missed heartbeat