CDD High Availability RabbitMQ Connection

Products

Continuous Delivery Director SAAS

Issue/Introduction

Using the steps outlined in the CDD Product documentation for setting up an Active-Active high availability environment, rabbitmq is still not showing any connections.

Environment

CDD 6.7-7
rabbitmq 3.6.6
RHEL 6.9
Tomcat8

Cause

It was determined that the documentation is missing a following step:

Create/Update the $CATALINA_BASE/bin/setenv.sh file with the following entry: export CDD_WS_BROKER_TYPE=relay

Connections were still not created after validating that the

Resolution

Make sure that all three required rabbitmq plugins have been installed:
1. rabbitmq_management
2. rabbitmq_stomp
3. rabbitmq_web_stomp
Create/Update the $CATALINA_BASE/bin/setenv.sh file with the following entry: export CDD_WS_BROKER_TYPE=relay
1. See note in the Additional Information section about Tomcat8 services on RHEL 6.9.
2. See note in the Additional Information section about how to verify that the CDD_WS_BROKER_TYPE environment variable was properly detected.

Setup rabbitmq using the configuration recommended for rabbitmq docker systems. After updating the configuration file restart rabbitmq. The product documentation recommends the following configuration file:

[{ rabbit, 
[ 
{
loopback_users,
[ ] 
},
{ 
log_levels,
%% The level can be one of 'none' (no events are logged), 'error' (only errors are logged), 'warning' (only errors and warning are logged), 'info' (errors, warnings and informational messages are logged), or 'debug' (errors, warnings, informational messages and debugging messages are logged).
[ 
%% channel - for all events relating to AMQP channels
{channel, debug}, 
%% connection - for all events relating to network connections
{connection, debug},
%% federation - for all events relating to federation (https://www.rabbitmq.com/federation.html)
{federation, debug},
%% mirroring - for all events relating to mirrored queues (https://www.rabbitmq.com/ha.html)
{mirroring,debug}
]
}
]
}].

Additional Information

Steps to setup active-passive and active-active high availability environment: https://docops.ca.com/ca-continuous-delivery-director/6-7/en/administration/configure-high-availability

Note:
An instance was found where Tomcat8 was installed as a service on RHEL 6.9. Prior to exposed to this instance, it was expected that the startup of tomcat8 (service and non-service) would use catalina.sh (internally) which calls setenv.sh. The setenv.sh is the file commonly used to specify custom configuration items. But in this instance there was no catalina.sh file and therefore setenv.sh is not being called. It was found that /etc/init.d/tomcat8 startup script was pointing to /usr/sbin/tomcat8. And /usr/sbin/tomcat8 is where tomcat gets started with the expected java command, classpath, etc.. So for point #2 in the Resolution section above we needed to set the "export CDD_WS_BROKER_TYPE=relay" line in the /usr/sbin/tomcat8 file, stop tomcat8, and start it.

Note:
When the CDD_WS_BROKER_TYPE environment is properly detected you should see the following log messages in the <Shared CDDHOME>/logs/cdd-server.<servername>.log file:
2018-09-13 16:18:56.965 [localhost-startStop-1] INFO c.c.rp.conf.SpringProfileInitializer - Found environment variable [CDD_WS_BROKER_TYPE] with value: relay
2018-09-13 16:18:56.966 [localhost-startStop-1] INFO c.c.rp.conf.SpringProfileInitializer - Using profile: websocket_relay_broker

Example log file messages of when it is not being properly picked up are:
2018-09-13 15:02:32.747 [localhost-startStop-1] INFO c.c.rp.conf.SpringProfileInitializer - No spring profile settings found, setting the default: websocket_simple_broker