RabbitMQ on-demand-broker VM in "Starting" State after upgrade to TAS / PCF 1.15

Products

• VMware RabbitMQ

Issue/Introduction

This article discusses a solution to the On-Demand Broker (ODB) "Starting" state loop, which occurs after upgrading to the RabbitMQ tile 1.15x in Tanzu Application Service (TAS), formerly Pivotal Cloud Foundry (PCF).

Resolution

The ODB Virtual Machine (VM) outputs the following "Starting" state loop log in bosh vms after upgrading RabbitMQ tile version 1.14.x to 1.15.x:

Note: Creating a new service instance with the Cloud Foundry Command Line Interface (cf CLI) commands will not work in this case.

Instance                        Process State AZ               IPs      VM CID                  VM Type Active  

on-demand-broker/abc20c5e-40c5-438e-806b-3595b91a3e87 starting    SB-Internal-Services-PCF-01-1 10.51.168.52 vm-329cf2d5-13a9-4dc3-9f06-6b6412027b7f micro  true  

The ODB logs in '/var/vcap/sys/log/broker' provide information regarding a connection timeout to port 8844:

[on-demand-service-broker] 2019/05/08 14:39:23.517905 Starting broker

[on-demand-service-broker] 2019/05/08 14:39:53.682293 error starting broker: Get https://1.2.3.4:8844/info: dial tcp 1.2.3.4:8844: i/o timeout

The above IP "1.2.3.4" belongs to the BOSH director. Port 8844 belongs to the CredHub instance, which runs inside BOSH director.

A firewall that is actively blocking traffic creates or produces communication problems between ODB and the CredHub instance. 

The tile in RabbitMQ version 1.15x stores on-demand Erlang cookies and admin passwords in CredHub. ODB must communicate with CredHub to work properly.

A rule or exception must be implemented which allows traffic to pass through the firewall from the ODB VM to the BOSH director IP and CredHub port 8844.

For more information, reference the RabbitMQ 1.15x release notes here.