Windows Servers May Hang on Shutdown or Restart after Windows Patching

Products

• Carbon Black EDR (formerly Cb Response)
• Carbon Black Hosted EDR (formerly Cb Response Cloud)

Issue/Introduction

After applying Windows updates or security patches, Windows may hang during the shutdown or restart of the system.

Environment

• EDR Windows Sensors: 7.3.0-7.3.1 & 7.4.0 & 7.4.1

Cause

Due to the amount of file and registry event modifications, these resources become locked by the sensor.     

Resolution

• These issues are addressed in Windows Sensor 7.3.2 (CB-39524)
• This issue has also been seen in Windows Sensor 7.4.0 and is resolved in 7.4.1 (CB-42042)
• If an upgrade is not possible, and you must remain on 7.3.2, then this Resolution can be followed instead. 
• The file contention issue can be avoided by temporarily disabling the collection of certain events during the time of file contention (which can lead to the hang). 

1.  Prior to modifying Group Settings, note which Event Collections are currently being used. 
2.  After applying Windows updates and prior to restarting, in Sensor Group Settings, disable "Binary module loads", "Binaries" and "Binary info".   Disabling these settings prevents the rehashing of the files, which will avoid the locking. 

3.  Reboot the endpoints as required by Windows.
4.  Re-enable the settings back to the original settings and remember to save the Group Settings.

Additional Information

• The hang is not wholly due to the updates, but due to changes made between EDR sensor versions 7.2.2 and 7.3.0 with regard to how the sensor locks files during the time it processes* them.
• The security updates result in overwriting core files (e.g., user32.dll) that are not usually modified, which reveals the overly-aggressive file locking.
• It's possible the machine will boot into safemode but not into normal windows.

* "processes" or "process the files": refers to copying to store directory, updating on-disk catalog, re-hashing the files (after them having been overwritten by the patch)