Windows servers experience a hang and only a reboot resolves the issue

Products

• Endpoint Protection

Issue/Introduction

Windows servers experience a hang and only a reboot resolves the issue.
You determined that the issue occurs with different versions of Symantec Endpoint Protection (SEP).

Environment

Symantec Endpoint Protection 12.1 and 14 (all versions with BASH 10)

Cause

This issue was determined to be a deadlock related to version 10 of the Behavioral Analysis And Security Heuristics (BASH) driver.
A memory dump analysis shows a large number of locked bhdrvx64.sys threads in a deadly embrace with a resource in the registry sub system (showing a shared resource nt!CmpRegistryLock), with a stack text similar to the following:

Child-SP          RetAddr           : Args to 
Child                                                           : Call Site
fffff880`11f29ea0 fffff800`01a67b62 : fffffa80`0d11eb50 fffffa80`0d11eb50 
fffff880`00000000 fffff800`0000000e : nt!KiSwapContext+0x7a
fffff880`11f29fe0 fffff800`01a78fef : 00000000`00000000 fffff800`01a42617 
fffff880`00000058 fffff800`01bf0e80 : nt!KiCommitThreadWait+0x1d2
fffff880`11f2a070 fffff800`01a523ca : fffffa80`0c34f700 fffff800`0000001b 
00000000`00000000 fffff880`026a4100 : nt!KeWaitForSingleObject+0x19f
fffff880`11f2a110 fffff800`01a789b1 : ffffffff`fd9da600 fffffa80`18aaa220 
fffff800`01bfa000 00000000`00000000 : nt!ExpWaitForResource+0xae
fffff880`11f2a180 fffff800`01d40f9c : fffff880`11f2a6d0 fffff8a0`00021220 
00000000`00000006 fffff880`01ad5048 : nt!ExAcquireResourceSharedLite+0x2c6
fffff880`11f2a1f0 fffff800`01d3d572 : fffff8a0`00021220 fffff880`11f2a6a0 
fffff880`11f2a728 fffff880`11f2a6d0 : nt!CmpBuildHashStackAndLookupCache+0x1bc
fffff880`11f2a5d0 fffff800`01d7219e : fffffa80`0c76e588 00000000`00000000 
fffffa80`0c76e3d0 00000000`00000000 : nt!CmpParseKey+0x5a7
fffff880`11f2a8c0 fffff800`01d72c86 : 00000000`00000000 fffff880`11f2aa40 
00000000`00000240 fffffa80`09781650 : nt!ObpLookupObjectName+0x784
fffff880`11f2a9c0 fffff800`01d4285c : fffff8a0`1e145b40 00000000`00000000 
00000000`00000000 fffff880`00000000 : nt!ObOpenObjectByName+0x306
fffff880`11f2aa90 fffff800`01d4da32 : fffff880`11f2ae48 00000000`00020019 
fffff880`11f2ade0 00000000`00000000 : nt!CmOpenKey+0x28a
fffff880`11f2abe0 fffff800`01a71413 : 00000000`00000000 00000000`00000000 
00000000`00000000 00000000`00000000 : nt!NtOpenKey+0x12
fffff880`11f2ac20 fffff800`01a6d9d0 : fffff880`0d321237 fffffa80`18f3ec50 
fffff880`0d32fe3e fffff880`11f2ae60 : nt!KiSystemServiceCopyEnd+0x13 
fffff880`11f2adb8 fffff880`0d321237 : fffffa80`18f3ec50 fffff880`0d32fe3e 
fffff880`11f2ae60 00000000`00000000 : nt!KiServiceLinkage
fffff880`11f2adc0 fffff880`0d2aae9d : 00000000`00000000 00000000`00000000 
fffff8a0`13262970 fffff880`0d3a3728 : BHDrvx64+0x97 
fffff880`11f2ae20 fffff880`0d282991 : fffff8a0`0000068a 00000000`00000000 
00000000`00000000 00000000`000007ff : BHDrvx64+0x1fd 
fffff880`11f2aee0 fffff880`0d2811b8 : 00000000`0000005c 00000000`00000000 
00000000`0000001a fffff880`11f2aff0 : BHDrvx64+0x1a1 
fffff880`11f2af50 fffff880`0d341396 : 00000000`0000001a fffff8a0`102968d8 
fffff8a0`12aa1310 fffff880`11f2b070 : BHDrvx64+0xb8 
fffff880`11f2af90 fffff880`0d341072 : 00000000`0000001a 00000000`00000000 
fffff880`0d3be2f8 fffff8a0`1a556e98 : BHDrvx64+0x46 
fffff880`11f2afd0 fffff880`0d340da7 : 00000000`000023ec fffff8a0`0f40001c 
00000000`00002300 fffff8a0`0f4640f6 : BHDrvx64+0x192 
fffff880`11f2b060 fffff880`0d339d4a : fffff8a0`0db69ff0 00000000`00006bc4 
fffff8a0`102968d0 fffff8a0`12aa1310 : BHDrvx64+0x3e7 
fffff880`11f2b110 fffff880`0d337084 : fffff8a0`0db6a030 fffff880`11f2b650 
fffff880`11f2b420 fffff8a0`185eda20 : BHDrvx64+0x29a 
fffff880`11f2b200 fffff880`0d337a1d : 00000000`00000000 fffff880`11f2b420 
fffff880`00000002 fffff8a0`185eda20 : BHDrvx64+0x74 
fffff880`11f2b250 fffff880`0d3df596 : 00000000`00000000 fffff8a0`0db5f001 
fffff880`11f2b650 fffff880`11f2b420 : BHDrvx64+0x25d 
fffff880`11f2b310 fffff880`0d22ebb7 : fffff880`11f2b650 fffff8a0`0db5f000 
fffff8a0`1935c010 fffff8a0`185eda20 : BHDrvx64+0x1e6 
fffff880`11f2b3b0 fffff880`0d22e441 : fffff8a0`0db5f020 00000000`00000000 
fffff880`11f2b650 00000000`00000000 : BHDrvx64+0x727 
fffff880`11f2b4f0 fffff880`0d2d381c : fffffa80`1ab50540 fffff8a0`185eda20 
fffff880`11f2b650 00000000`00000000 : BHDrvx64+0x181 
fffff880`11f2b570 fffff880`0d2d3715 : fffff880`11f2b650 fffff880`0178f76f 
00000000`0bac1c01 fffff880`00000000 : BHDrvx64+0x2c 
fffff880`11f2b5a0 fffff880`0d2d1e9b : fffffa80`1ab50540 fffff8a0`13d84408 
00000000`00000000 fffff880`11f2b9f0 : BHDrvx64+0xc5 
fffff880`11f2b600 fffff880`0d2d192a : 00000000`00000000 00000000`00000000 
00000000`10000004 fffff880`035e9deb : BHDrvx64+0xab 
fffff880`11f2b7b0 fffff880`0d2d089f : 00000000`00000001 fffffa80`0cfa0b90 
fffff880`11f2b8c8 00000000`00002ad8 : BHDrvx64+0x7a 
fffff880`11f2b810 fffff880`01790067 : 00000000`00000000 00000000`00000072 
fffffa80`0a570040 fffff800`01a78fef : BHDrvx64+0x15f 
fffff880`11f2b850 fffff880`01791329 : fffff880`11f2b900 00000000`00000012 
00000000`00000000 00000000`0031ec00 : fltmgr!FltpPerformPreCallbacks+0x2f7
fffff880`11f2b950 fffff880`0178f6c7 : fffffa80`0d087890 fffffa80`0a570040 
fffffa80`0a05f780 fffffa80`09a1d070 : fltmgr!FltpPassThrough+0x2d9
fffff880`11f2b9d0 fffff800`01d7fb8f : fffffa80`0d087890 fffffa80`0da5a060 
00000000`00000000 fffffa80`09a1d070 : fltmgr!FltpDispatch+0xb7
fffff880`11f2ba30 fffff800`01d6db3e : 00000000`00000000 fffffa80`09a1d040 
fffff8a0`03c667f0 fffffa80`0dc5c060 : nt!IopCloseFile+0x11f
fffff880`11f2bac0 fffff800`01d6d7af : fffffa80`09a1d040 fffffa80`00000001 
fffff8a0`15b094f0 00000000`00000002 : nt!ObpDecrementHandleCount+0x8e
fffff880`11f2bb40 fffff800`01d6ded4 : 00000000`00000064 fffffa80`0da5a060 
fffff8a0`15b094f0 00000000`00000064 : nt!ObpCloseHandleTableEntry+0xaf
fffff880`11f2bbd0 fffff800`01a71413 : fffffa80`0d11eb50 fffff880`11f2bca0 
00000000`00000002 fffffa80`0c0c5950 : nt!ObpCloseHandle+0x94
fffff880`11f2bc20 00000000`76efbc2a : 00000000`00000000 00000000`00000000 
00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 
00000000`0031eba8 00000000`00000000 : 00000000`00000000 00000000`00000000 
00000000`00000000 00000000`00000000 : 0x76efbc2a

Resolution

This issue was resolved in the BASH 11 update, which was published to all Enterprise customers on November 29, 2016.