crowdstrike bosh add detects drift and prevents application container from staging on diego cell

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

You will see various different types of errors depending on what buildpack your app uses to build.  During staging the buildpack process will get killed and exit at random points of the staging process.  The most common symptom across all buildpacks will be "Exit Status 223".    Crowdstrike real time drift detection feature maybe targeting a single diego cell out of 100 and you might see all new app containers crash on this one cell all the time.  While other cells are able to successfully start containers. 

 

Example from nodejs buildpack during build dependencies step gets killed by crowdstrike

**ERROR** Unable to build dependencies: signal: killed 
Failed to compile droplet: Failed to run all supply scripts: exit status 14 
Exit status 223 

Environment

This problem was first detected with the following tile versions

Crowdstrike Tile Version 1.4.0 

Tanzu Platform Cloud Foundry 4.0.21

Resolution

Workaround

When the container crashes a log message like the one below will provide which diego cell ID was involved in the container crash.  in the example below cell id is "22222222-2222-2222-2222-222222222222".  If you find the crash symptoms are consistently present on a single cell or a small handful of cell then proceed to ssh into the diego cells and run "monit stop all".  This will remove the cell from future container auctions and allow apps to get placed on cells not impacted by crowdstrike.  

Cell 22222222-2222-2222-2222-222222222222 destroying container for instance 11111111-1111-1111-1111-111111111111

 

Another possible workaround would be remove crowdstike addon or monit stop the crowdstrike agents and falcon sensor on the diego cell. 

 

Resolution

Contact Crowd strike vender support for solution