Health status error on "RabbitMQ server is not running"

Products

• Malware Analysis Software - MA

Issue/Introduction

1. Malware Analysis Appliance system status on management UI shows red with health error on "RabbitMQ server is not runing"

2. /var/log/syslog shows health stats error exception page similar to below example:

'---------------------------------------- START OF EXCEPTION----------------------------------------'
Sep 21 10:43:17 (none) health[3036]: ''
Sep 21 10:43:17 (none) health[3036]: "  LOG MSG         : 'Failed to fetch rabbitmq status'"
Sep 21 10:43:17 (none) health[3036]: '  EXCEPTION CLASS :exceptions.AttributeError'
Sep 21 10:43:17 (none) health[3036]: "  EXCEPTION MSG   :''NoneType' object has no attribute 'split''"

<truncated for brevity>

3. Rabbitmq service shows error status similar to below example:

  root@(none):~# rabbitmqctl status
  Status of node 'rabbit@xxx' ...
  Error: unable to connect to node 'rabbit@xxx': nodedown
  diagnostics:

4. Rebooted and after coming back up, cannot login to the management UI.

Cause

The rabbitmq service isn't starting and in most cases the system is having a corrupted mnesia database in /var/lib/rabbitmq/mnesia directory

Resolution

Deleting the rabbitmq 'mnesia' data store should resolve this issue (mnesia data store will repopulate as needed)

* Log in via SSH as 'g2' (or log onto the console)
* Run below and provide the 'g2' password when needed

g2@(none):~$sudo supervisorctl stop all <-- this can take a few minutes
g2@(none):~$sudo /etc/init.d/rabbitmq-server stop
g2@(none):~$sudo rm -rf /var/lib/rabbitmq/mnesia/*

Then reboot the box; rabbitmq should start properly and the dependent services should as well. Upon boot up, re-check the rabbitmq service using "sudo rabbitmqctl status" or "/etc/init.d/rabbitmq-server status"

1. Make sure normal output as below shown:
g2@(none):~# sudo rabbitmqctl status
  Status of node 'rabbit@xxx' ...
  [{pid,...},
   {running_applications,[{rabbit,"RabbitMQ","3.6.12"},
                          {os_mon,"..."},
                          {sasl,"..."},
                          {mnesia,"..."},
                          {stdlib,"..."},
                          {kernel,"..."},
  {os,"..."},
  {erlang_version,"..."},
  {memory,"..."}}]
<truncated for brevity>

2. Make sure the process is running similar to below output: (There should be 5 processes running as the user rabbitmq.)

g2@(none):~# ps -ef | grep rabbit
rabbitmq  1866     1  0 10:49 ?        00:00:00 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
rabbitmq  1893     1  0 10:49 ?        00:00:17 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@(none) -boot /var/lib/rabbitmq/mnesia/rabbit@(none)-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@(none).log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@(none)-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@(none)"

rabbitmq  2031  1893  0 10:49 ?        00:00:00 /usr/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
rabbitmq  2036  1893  0 10:49 ?        00:00:00 inet_gethost 4
rabbitmq  2037  2036  0 10:49 ?        00:00:00 inet_gethost 4