EDR Cluster: Minion startup fails at Rabbitmq Error: cannot_delete_plugins_expand_dir

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Message in /var/log/cb/rabbitmq/startup_err

Starting rabbitmq
{"init terminating in do_boot",{error,{cannot_delete_plugins_expand_dir,["/var/cb/data/rabbitmq/mnesia/rabbit@CB-SERVER-CLUSTER-HEAD-NODE-plugins-expand",{cannot_delete,"/var/cb/data/rabbitmq/mnesia/rabbit@CB-SERVER-CLUSTER-HEAD-NODE-plugins-expand/rabbitmq_management-3.7.14",eacces}]}}}

 

Environment

• EDR Server: 7.0.1
• Clustered Environment

Cause

Corrupted mnesia directory and interferance from epmd daemon

Resolution

1. Stop cluster

   • /usr/share/cb/cbcluster stop

2. Confirm services are stopped on each node of the cluster
   1. Check node service status

      • service cb-enterprise status

   2. Stop any running services in ascending order

      • service <cb-service> stop

   3. Kill any other cb processes

      • killall -KILL -u cb

   4. Confirm cb services are stopped

      • ps -ef | grep cb

3. Move the mnesia directory on each node of the cluster

   • mv /var/cb/data/rabbitmq/mnesia /var/cb/data/rabbitmq/mnesia.bkp

4. Start cluster from primary node

   • usr/share/cb/cbcluster start

5. Remove the mnesia backup

   • rm -rf /var/cb/data/rabbitmq/mnesia.bkp

Additional Information

https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-reset-Mnesia-for-RabbitMQ/ta-p/64848
https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-Restart-Server-Services/ta-p/41294