CB Response: Services do not Start After Upgrading to 6.4.0

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

After upgrading to 6.4.0 the server services do not complete start-up

• RabbitMQ appears to start but fails at Solr
• /var/log/cb/supervisord/supervisord.log shows this error milliseconds after rabbitmq startup

  cb-rabbitmq (exit status 1; not expected)

• /var/log/cb/rabbitmq/rabbit\@CB-SERVER-CLUSTER-HEAD-NODE.log

  2019-07-02 08:50:53.892 [info] <0.8.0> Log file opened with Lager
  2019-07-02 08:50:54.720 [error] <0.8.0> Cluster upgrade needed but other disc nodes shut down after this one.
  Please first start the last disc node to shut down.
  
  Note: if several disc nodes were shut down simultaneously they may all
  show this message. In which case, remove the lock file on one of them and
  start that node. The lock file on this node is:
  
   /var/cb/data/rabbitmq/mnesia/rabbit@CB-SERVER-CLUSTER-HEAD-NODE/nodes_running_at_shutdown

Environment

• Carbon Black Response Server: 6.4.0
• Clustered Environment

Cause

RabbitMQ has been updated to 3.7 and nodes do not come back up in the order they went down at service stop.

Resolution

1. Services may come up on a second attempt

   /usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

2. If the services do not come up on the second attempt follow these steps
   1. Stop the Services completely

      /usr/share/cb/cbcluster stop

   2. Remove the following file on any node, this example is from the master

      rm -f /var/cb/data/rabbitmq/mnesia/rabbit\@CB-SERVER-CLUSTER-HEAD-NODE/nodes_running_at_shutdown

   3. Start RabbitMQ on that node

      service cb-rabbitmq start

   4. Stop RabbitMQ on the node

      service cb-rabbitmq stop

   5. Start all services

      /usr/share/cb/cbcluster start

Additional Information

• Configuration changes are currently being evaluated for resolution
• This will only need to be done the first time after the upgrade