EDR: Windows endpoints BSOD running 7.x sensor with very large "C:\Windows\CarbonBlack\store\catalog" file

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

7.2.0-win endpoints were having BSOD issues when OS Watchdog service times out reading the C:\Windows\CarbonBlack\store\catalog file. This will generate a memory.dmp file that 

Environment

• EDR Windows Sensor: 7.x through 7.2.1
• C:\Windows\CarbonBlack\store\catalog file excessive growth

Cause

This crash happened because a kernel watchdog decided that some process was taking too long. The system is booting and loading cbk7.sys. Cbk7.sys starts filtering, attaches to the system volume, and tries to set up the carbonblack file store (local binary repo). The file store helper is reading the catalog file. The watchdog timer fires, which causes a DPC to interrupt the thread.

Resolution

• CB-35610 was fixed from EDR sensor version 7.2.2 onwards
• Workaround: Remove the C:\Windows\CarbonBlack\store\catalog file on affected endpoints when they grow larger than expected sizes.

Additional Information

• Workaround can be applied either before upgrading to 7.2.0-win OR in safe mode if 7.2.0 is installed with Tamper Protection setup per Sensor Group and the endpoints are encountering the BSOD. 
• The impacted sensor versions (7.2.0 and 7.2.1) have reached end of support: https://docs.vmware.com/en/VMware-Carbon-Black-EDR/services/cb-edr-oer-win-server-sensor/GUID-2DE14E48-F61F-4D5C-A167-530A55CC2559.html#:~:text=22%20February%202023-,7.2.1,-3%20June%202021
• ACE Engineering has created CB-35610 to have future driver versions check the size of the catalog file prior to reading it starting in 7.3.0-win sensor.
• Catalog file size is subjective, but the example case we found catalog files over 1.7GB+, which was taking a very long time to read during OS startup.