Rabbitmq service fails to start on vIDM node

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• vIDM admin UI shows rabbitmq service not running successfully
  • 
• You have verified there is plenty of disk space available on the system. To check run the following command

df -h

• The VMware Identity Manager upgrade halts because the pre-upgrade check fails to communicate with an inactive local RabbitMQ service node.
• Following the updated package installation, the rabbitmq-server.service completely fails to start, throwing a hard exit error in the system upgrade logs.
  

  • RabbitMQ version is 3.10.7. Running mandatory pre-upgrade step...
Error: unable to perform an operation on node 'rabbitmq@<REDACTED_HOSTNAME>'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

    DIAGNOSTICS
===========
attempted to contact: ['rabbitmq@<REDACTED_HOSTNAME>']
rabbitmq@<REDACTED_HOSTNAME>:
  * connected to epmd (port 4369) on <REDACTED_HOSTNAME>
  * epmd reports: node 'rabbitmq' not running at all
                  no other nodes on <REDACTED_HOSTNAME>
  * suggestion: start the node

    Current node details:
 * node name: 'rabbitmqcli-889-rabbitmq@<REDACTED_HOSTNAME>'
 * effective user's home directory: /root
 * Erlang cookie hash: <REDACTED_SECRETS>

    Stopping RabbitMQ service before upgrade to 3.11.18...
Cleaning up old logs in /var/log/rabbitmq...
Updating Rabbitmq to 3.11.18 using /usr/local/horizon/rpms/rabbitmq-server-3.11.18-1.ph3.noarch.rpm...
Installing/Updating /usr/local/horizon/rpms/rabbitmq-server-3.11.18-1.ph3.noarch.rpm...
Preparing...                          ########################################
Updating / installing...
rabbitmq-server-3.11.18-1.ph3         ########################################
Cleaning up / removing...
rabbitmq-server-3.10.7-5.ph3          ########################################
Installation/Update of /usr/local/horizon/rpms/rabbitmq-server-3.11.18-1.ph3.noarch.rpm is successful.
Starting RabbitMQ service...
Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
ERROR: RabbitMQ service is not active after upgrade to 3.11.18.
Please make sure rabbitmq is upgraded to 3.11.18 to proceed with upgrade

• Running the command 'service rabbitmq-server status' or 'journalctl -xe' shows the following error:

Starting broker

Boot failed

Error during startup: error, schema_integrity_check_failed, table_missing, rabbit_index_routing

or

Error: feature flags: ‘classic_queue_type_delivery_support’ required feature flat not enabled it must be enabled before upgrading RABBITMQ

Failed to initialize feature flags registry: disabled_required_feature_flag, classic_queue_type_delivery_support

Boot failed

Environment

vIDM 3.3.7

Resolution

To resolve this issue follow the steps below:

NOTE - If there are disk space issues on the system please address those before attempting the steps below. A full partition on the system can also cause the rabbitmq service fail to start 

1. Take a backup/snapshot of the vIDM appliance(s) having issues starting rabbitmq service
2. Run the following commands in order -

systemctl stop rabbitmq-server.service

rm -rf /db/rabbitmq/data

service horizon-workspace restart

systemctl start rabbitmq-server.service

3. Log back into the vIDM admin UI and verify the rabbitmq service is now started successfully