Application connectivity issues after reinstalling Pre-Provisioned RabbitMQ Service

Products

• VMware RabbitMQ

Issue/Introduction

This article is applicable to all versions of the Pre-Provisioned RabbitMQ Service

After reinstalling the Pre-Provisioned RabbitMQ Service, existing Applications started timing out, while trying to connect to RabbitMQ servers. To be specific, the applications are not able to connect to port 5671.

Resolution

As part of the Pre-Provisioned RabbitMQ Service installation, the static IPs need to be assigned to HAProxy, RabbitMQ Server, and Service Broker. If any of these fields are left blank, BOSH allocates an IP address. The BOSH-assigned dynamic IP address for HAProxy will be different than the static IP address the existing clients are using. To resolve the issue, specify the correct static IP addresses for Pre-Provisioned RabbitMQ and restart the applications.

Troubleshooting Steps:

• Check the status of the Rabbitmq cluster in the management UI. 
• If the status of the RabbitMQ nodes is fine, then check if applications are able to connect to port 5671
• If the applications are not able to connect to port 5671, then check the status of the SSL/TLS connection between the Server: and the clients, using the OpenSSL s_client command which establishes a TLS/SSL connection to a remote server and checks, if a server is properly configured for SSL/TLS and to see the certificate chain presented by the server.
• If the SSL/TLS connection, between the Server: and the clients, is working, then check the application side logs to identify the HAProxy IP that application logs are using to connect to the server.
• Check if the broker and the HAProxy IPs assigned for the Pre-Provisioned RabbitMQ Service are correct and are identical to what the clients are using.
• If client IPs do not match the HAProxy IP for the Pre-Provisioned RabbitMQ Service, then provide the correct static HAProxy IP in the Pre-Provisioned RabbitMQ Service and restart the application.

References:

Installing and Configuring the Pre-Provisioned RabbitMQ Service
How to Specify Static IP Addresses for Pre-Provioned RabbitMQ VM
OpenSSL s_cleint