VIO rabbitmq not starting error while waiting for Mnesia tables

Products

• VMware Integrated OpenStack

Issue/Introduction

• rabbitmq pod failed to start and kept crashing

osctl logs -f rabbitmq1-rabbitmq-0

...
2019-12-06 08:14:11.626 [warning] <0.250.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2019-12-06 08:14:11.657 [info] <0.250.0> Waiting for Mnesia tables for 30000 ms, 8 retries left
...

 

Environment

6.x, 7.x

Cause

Unclean rabbitmq shutdown

Resolution

For VMware Integrated OpenStack 6.x:

1. Edit rabbitmq1-rabbitmq statefulset in insert 'rabbitmqctl force_boot'

osctl edit statefulset rabbitmq1-rabbitmq

change

      containers:
      - command:
        - /tmp/rabbitmq-start.sh

to

      containers:
      - command:
        - bash
        - -c
        - |
          rabbitmqctl force_boot
          /tmp/rabbitmq-start.sh

2. Find existing rabbitmq pods.

root@oms [ ~ ]# osctl get pod | grep  rabbitmq1-rabbitmq
rabbitmq1-rabbitmq-0                                              1/1     Running     0          8h
rabbitmq1-rabbitmq-1                                              1/1     Running     0          8h
rabbitmq1-rabbitmq-2                                              1/1     Running     0          8h

3. Delete the pods from Step 2.

osdel pod <rabbitmq pod>

4. Wait for rabbitmq pod(s) to be re-created and OpenStack Deployment State to say RUNNING.

watch -n2 'viocli get deployment'
 

For VMware Integrated OpenStack 7.x:

1. Edit the custom resource for rabbitmq with viocli update rabbitmq and insert the following:

spec:
  conf:
    force_boot: true

2. Wait for rabbitmq pod(s) to be re-created and OpenStack Deployment State to say RUNNING with the following command: ​​​​

watch -n2 'viocli get deployment'

Additional Information

Will be able to clear mnesia state to restart; there's nothing really valuable there.