Update of RabbitMQ for Pivotal Platform causes Pivotal Platform push notifications downtime

Products

• VMware RabbitMQ

Issue/Introduction

Symptoms:

After upgrading the PCF RabbitMQ tile from 1.9.5 to 1.9.6 with the Push Push Notifications Service errand disabled, the Push Notifications service stop functioning.

In the push-API application logs, you will find something similar to

OUT [ERROR] ..... io.pivotal.push.service.queue.impl.RabbitMQAbstractQueue - [TENANT ...] - 
Failed to enqueue: attempted #1 with exception {}

Environment

Cause

The upgrade of the PCF RabbitMQ tile will cause the existing Push Notifications service to lose connectivity to Rabbit. Prior to PCF Push Notifications 1.9.3, once this connectivity is lost, the Push Notifications service will not automatically try to reconnect without being restarted.

Resolution

There are 3 distinct ways to resolve or avoid this problem:

• Restart App: Manually restart the "push-API" application via the CF CLI.
• Upgrade Push Notifications: Upgrade to a version of PCF Push Notifications greater than 1.9.2.
• Avoidance: Have the Push Push Notifications Service errand enabled when you are upgrading PCF RabbitMQ.