How to troubleshoot a RabbitMQ Cluster for Mobility Suite.

Products

• Mobility Suite

Issue/Introduction

 

Resolution

The following commands can be run on either the Rabbit Master or Rabbit Slave:
sudo /usr/sbin/rabbitmqctl cluster_status

The "running" list should have two nodes displayed.
sudo rabbitmqctl -p <vhost> list_policies

There should be two different entries for this command which checks the policies that are applied to the cluster.

*Note* - Our documentation listed in HOWTO107258 explains that the set_permissions command should be used with single quotes. This is not the case and should use the following example syntax:
sudo rabbitmqctl set_permissions -p vhost username ".*" ".*" ".*"

The celery.log files in Mobility Suite will display 403 errors if the permissions for RabbitMQ are not setup in this exact format. The celery services will also appear to start, but shut down immediately afterwards.

If the On-Premise installation of Mobility Suite 5.2 results in a failure immediately after continuing past the Configurator webpage and displays the following message, then the rabbitmq daemon needs to be turned off if a trial installation was configured:
'Failed to provision software. Please check /var/log/nukona/load_settings.log for details'

The rabbitmq daemon can be turned off by executing the following command from the Terminal:
sudo -u rabbitmq rabbitmqctl stop

The installation can proceed as expected if rabbitmq is turned off. The setup.sh script will not need to be re-run. The configurator process can be resumed rabbitmq has been turned off by pressing the 'Close' button where the error was displayed and pressing the 'Resume' button that appears once the error is cleared.

The rabbitmq daemon can be started by running the following command from the Terminal:
rabbitmq-server

The following command can be run from the Terminal to list the permissions to the target vhost for all users:
rabbitmqctl list_permissions -p <vhost>

To check if the rabbitmq service is running the following command can be run:
rabbitmqctl status

To check if the rabbitmq-server is set to start on reboot (2,3,4, & 5 values) the following command can be run:
chkconfig --list

The following commands can be set to have rabbitmq-server start on reboot if it is not already set to 'on':

chkconfig --add rabbitmq-server
​chkconfig --level 2345 rabbitmq-server on

The following commands can be used to check the format of the hostname and edit it if necessary (127.0.0.1 and ::1 should resolve to the hostname):
vi /etc/hosts
vi /etc/sysconfig/network 

To check if the master and slave servers' .erlang.cookie checksums match, the follow command can be run on both servers:
md5sum /var/lib/rabbitmq/.erlang.cookie

The following log files can be collected for further troubleshooting purposes:
1.  The startup_error and startup_log files in the /var/log/rabbitmq​ directory.
2.  The erl_crash.dump file located in the /var/lib/rabbitmq directory.

 

If the below errors are shown in the /var/log/nukona/celery.log and /var/log/nukona/appstore.log files on the Mobility Suite server, then the password will need to be changed to match what is listed in the /usr/local/nukona/etc/settings.cfg file for the RabbitMQ entry:

Connection error: [Errno 111] Connection refused.

Connection error: [Errno 104] Connection reset by peer.

In order to change the password for the rabbitmq server, the following command can be used:

# rabbitmqctl change_password {USERNAME.EN_US} {NEWPASSWORD.EN_US}

Example:

# rabbitmqctl change_password rabbitadmin P@ssw0rd