CB ThreatHunter: DPC_WATCHDOG_VIOLATION BSOD and Performance Issues

Products

• Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

• Endpoints Blue Screen/Crash
• High CPU usage

Environment

• CB ThreatHunter Web Console: All Versions
• CB PSC Sensor: 3.4.x.x through 3.5.0.1523

Cause

The cause of this issue is due to the use of the spin lock mechanism adversely affecting the performance of queuing events on high activity endpoints

Resolution

Upgrade impacted endpoints to sensor version 3.5.0.1590 or higher

Additional Information

The following information from analyzing a crash dump file indicate this issue is being experienced

• Bugcheck to look for DPC_WATCHDOG_VIOLATION (133)
• Partial call stack with ctifile calling KeAcquireSpinLockRaiseToDpc

22: kd> !analyze -v
*******************************************************************************
*
Bugcheck Analysis *
*
*******************************************************************************
DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000000, A single DPC or ISR exceeded its time allotment. The offending
component can usually be identified with a stack trace.
Arg2: 0000000000000501, The DPC time count (in ticks).
Arg3: 0000000000000500, The DPC time allotment (in ticks).
Arg4: 0000000000000000, cast to nt!DPC_WATCHDOG_GLOBAL_TRIAGE_BLOCK, which contains
additional information regarding this single DPC timeout
fffff880`034b58e8 fffff803`da4f8db5 : 00000000`00000133 00000000`00000000 00000000`00000501 00000000`00000500 : nt!KeBugCheckEx
fffff880`034b58f0 fffff803`da4f88c1 : fffffa80`61e31401 fffff880`02efa180 00000000`034b0100 fffff880`0347d180 : nt!KeAccumulateTicks+0x415
fffff880`034b5980 fffff803`da441df4 : ffffffff`ffd184d0 fffffa80`6114a502 00000000`00000000 00000000`00000304 : nt!KeUpdateRunTime+0x51
fffff880`034b59b0 fffff803`da63b2a6 : 00000000`02288f6f 00000000`023c3610 fffffa80`6114a520 fffff880`034b5b70 : hal!HalpTimerClockInterrupt+0x50
fffff880`034b59e0 fffff803`da4fb560 : ffffffff`5f636357 00000000`0000001c 00000000`00000000 00000000`00000801 : nt!KiInterruptDispatchNoLockNoEtw+0xe6
fffff880`034b5b70 fffff803`da59b3a1 : fffff880`0347d180 fffff803`da59ca02 fffff880`0347d180 fffffa80`e82e0c70 : nt!KxWaitForSpinLockAndAcquire+0x20
fffff880`034b5ba0 fffff803`da4fb533 : 00000000`00000002 fffff880`034b5df4 fffff880`034b5c90 00000000`00000304 : nt!KiAcquireSpinLockInstrumented+0x65
fffff880`034b5bf0 fffff880`8445e65e : fffff880`034b5c78 00000000`002822f3 fffffa80`879c6a10 fffff880`034b5cc0 : nt!KeAcquireSpinLockRaiseToDpc+0x43
fffff880`034b5c20 fffff880`84460b0d : 00000000`00000004 fffff880`84481f2d 00000000`00000008 fffffa80`e598e120 : ctifile+0x165e
fffff880`034b5c50 fffff880`8447b5dd : fffffa80`e17d8a90 fffffa80`e598e010 00000000`00000000 00000000`00000000 : ctifile+0x3b0d
fffff880`034b5ca0 fffff880`8447b6fd : fffffa80`e17d8a90 00000000`00000008 fffff880`034b5df0 00000000`00000610 : ctifile+0x1e5dd
fffff880`034b5ce0 fffff880`8466fecb : fffffa80`e17d8a90 00000000`00000008 fffff880`034b5df0 00000000`00000000 : ctifile+0x1e6fd
fffff880`034b5d10 fffff880`846701c9 : 00000000`00000002 fffff880`034b5e80 00000000`00001b28 00000000`00000000 : ctinet+0x1ecb
fffff880`034b5d80 fffff880`0154a0c3 : fffffa80`61792f50 fffffa80`61e31250 00000000`00000000 00000000`00000000 : ctinet+0x21c9
fffff880`034b5f10 fffff880`01544e70 : fffffa80`da5c0030 fffff880`034b6588 fffffa80`00000000 fffffa80`6cf9fdf0 : NETIO!ProcessCallout+0x363
fffff880`034b5fe0 fffff880`01544684 : 00000000`00000000 fffff880`034b6588 00000000`00000002 fffff880`034b62b0 :