Push Notification Service is Unable to Reconnect Queues after RabbitMQ Restarts

Products

• VMware Tanzu RabbitMQ

Issue/Introduction

Symptoms:

After RabbitMQ outages like restart or upgrade or some other issues, push-analytics and push-api cannot connect back to RabbitMQ instance used for queuing push notifications.

push-analytics app panics with an error:

org.springframework.amqp.rabbit.listener.QueuesNotAvailableException:
 Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.

push-api app panics with an error: 

[OUT] [WARN] 1196346734 23 Jan 2018 16:12:19.196 +0000 [push-worker-pool-ios-3]
 io.pivotal.push.jobs.workers.RabbitMQAbstractWorker - [SYSTEM] Rabbit seems to be down,
 waiting for reconnect 
 [push-worker-pool-android-baidu-0] io.pivotal.push.jobs.workers.RabbitMQAbstractWorker
 - [SYSTEM] Rabbit seems to be down, waiting for reconnect 
 [dispatch-worker-pool-6] io.pivotal.push.jobs.util.FixedWorkerRestartableThreadPoolExecutor
 - [SYSTEM] Restarting worker thread: RabbitMQDispatchWorker, 1792240021 
 [dispatch-worker-pool-6] io.pivotal.push.jobs.workers.RabbitMQAbstractWorker
 - [SYSTEM] Rabbit seems to be down, waiting for reconnect 
 [APP/PROC/WEB/0] [OUT] com.rabbitmq.client.AlreadyClosedException:
 connection is already closed due to connection error;
 cause: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds

Impact

This affects the functionality of push messages and can cause loss of small to a large number of push notifications.

Environment

Cause

This is a known issue with Push Notification service with the push notifications service tile.

 

Resolution

Restart the push-api and push-analytics app to reconnect to the RabbitMQ instance.

1. Login to cf cli
2. cf target -o system -s push-notifications
3. cf restart push-api
4. cf restart push-analytics

Permanent Fix

Upgrade the Push Notification Service tile to version 1.9.4.