All RabbitMQ VMs down after upgrade with Crowdstrike plugin

Products

VMware Tanzu RabbitMQ VMware RabbitMQ Pivotal RabbitMQ

Issue/Introduction

After adding the Crowdstrike tile to Opsman and applying changes to the RabbitMQ tile, RabbitMQ VMs show in "failing" state in bosh vms command output
The RabbitMQ OnDemand Broker (ODB) VM shows "stopped" state in bosh vms command output
Restarting the OnDemandBroker VM will fail during the pre-start job with errors like:

Error: Action Failed get_task: Task xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx result: 1 of 7 pre-start scripts failed: Failed Jobs: falcon-linux-sensor
From an SSH session into the ODB VM, we can see that the Falcon sensor is already installed using systemctl commands:

# sudo systemctl status falcon-sensor
From an SSH session into the ODB VM, monit summary commands indicate that only the system job (used to install packages) is running, no other monit processes are running

Environment

This issue was observed on an environment using Opsman 3.0.22 with RabbitMQ v2.4.2 using the Crowdstrike tile in Opsman.

Cause

When the falcon-sensor is manually installed on the ODB and RabbitMQ VMs, and the Crowdstrike tile is subsequently added to Opsman, the previously installed falcon-sensor blocks installation of the falcon-linux-sensor via Bosh pre-start scripts. Because the package can't be installed via Bosh, the pre-start script cannot complete for the falcon-linux-sensor, which prevents the secondary services from starting and leaves the VM in a "stopped" state. Because the ODB is required by the RabbitMQ VMs, and it is upgraded first, it will present in a "stopped" state, while the RabbitMQ VMs depending on ODB communication will go into "failed" state as reported by Bosh.

Resolution

The manually installed falcon-sensor service must be removed from VM's prior to applying changes from Opsman Installation Dashboard. The following commands can be used to remove the falcon-sensor from an SSH session to the ODB or RabbitMQ VMs:

# sudo apt-get remove --purge falcon-sensor

For any VM's that an upgrade was already attempted on that are stuck in "stopped" state, a restart will be required to install the falcon-sensor again.

# bosh -d $SERVICE_DEPLOYMENT restart ODB_VM

If the upgrade has been applied to the RabbitMQ VMs and one appears in "stopped" state, the RabbitMQ VM will have to be restarted with the --skip-drain flag due to the following known issue, which prevents RabbitMQ VMs from starting services. Once the VM is back into "running" state, it will have reinstalled the original falcon-sensor, so the falcon-sensor will have to be removed again prior to applying changes from Opsman. This ensures Bosh can successfully reconcile the new falcon-linux-sensor during deployment:

# bosh -d $SERVICE_DEPLOYMENT restart RABBITMQ_VM --skip-drain

SSH into the RabbitMQ VM and remove the newly installed Falcon driver prior to applying changes from Opsman:

# sudo apt-get remove --purge falcon-sensor