search cancel

High CPU usage seen on systems with Endpoint Protection and Endpoint Detection and Response

book

Article ID: 238115

calendar_today

Updated On:

Products

Endpoint Protection Endpoint Detection and Response

Issue/Introduction

High CPU utilization is seen on server where the Symantec Endpoint Protection (SEP) client is installed. 
Due to this the system is going in a hang state and are unable to perform any activity on that server.
Symantec Endpoint Detection and Response (SEDR) is also present in the environment.

Cause

The high CPU usage is caused by the IPS feature. You can try disabling Netstat events in IDSVia64.sys to confirm. 
IPS submissions must be disabled in SEPM and if applicable, Netstat events need to be disabled in EDR. 

WPP logging, symtdi-0000-SEPAutoTraceSession_20220224_225021.log shows many connections over port 389. The driver message queue is full because user mode can't process them fast enough.

Below is the dump analysis:

2: kd> !mex.t ffff95884f033800
Process                             Thread                       CID       TEB               UserTime KernelTime ContextSwitches Wait Reason Time State
ccSvcHst.exe *32 (ffff958849c4d800) ffff95884f033800 (E|K|W|R|V) c64.1218  0000000000554000 9m:00.750  4m:51.984           44181 UserRequest    0 Running on processor 1

 # Child-SP         Return           Call Site                                                      
 0 000000000568fa0c 000000006e57d29f IDSxpx86!__SEH_prolog4+0x25                                    
 1 000000000568fa40 000000006e57d4dc IDSxpx86!ccLib::CCriticalSection::Lock+0x7d                    
 2 000000000568fa50 000000006e57d508 IDSxpx86!ccLib::CSingleLock::Lock+0xf                          
 3 000000000568fa60 000000006e580b7d IDSxpx86!ccLib::CSingleLock::CSingleLock+0x1d                  
 4 000000000568fa74 00000000738a37ed IDSxpx86!ccSym::CMemoryStreamImpl::Read+0x20                   
 5 000000000568faa0 00000000738aebc2 ccLib!ccSym::CStreamArchive::ReadEx+0x72                       
 6 000000000568fab8 00000000738ad178 ccLib!ccLib::CArchive::Read+0xaa                               
 7 (Inline)         ---------------- ccLib!ccLib::CArchive::Read+0x14                               
 8 000000000568fae4 00000000738a89d5 ccLib!ccLib::CArchive::Read+0x18                               
 9 000000000568fb00 000000007388b187 ccLib!ccSym::CValueCollection::CValue::Load+0x32               
 a 000000000568fb38 000000007389ad51 ccLib!ccSym::CKeyValueCollection::Load+0x170                   
 b 000000000568fb74 000000006e4c2caa ccLib!ccSym::CSerialize::Load+0x62                             
 c 000000000568fbd4 000000006e46acb8 IDSxpx86!CMessageSerializer::Load+0x13a                        
 d 000000000568fc04 000000006e4a01c2 IDSxpx86!IDSxpx86::CIDSMessageChannelManager::Deserialize+0x18 
 e 000000000568fc1c 000000006e49fe9f IDSxpx86!IDSxpx86::CUserChannel::getNextMessage+0x82           
 f 000000000568fc44 000000006e57b500 IDSxpx86!IDSxpx86::CUserChannel::Run+0x1cf                     
10 000000000568fc80 000000006e57b2b7 IDSxpx86!ccLib::CThread::ThreadProc+0x123                      
11 000000000568fca0 000000006e57b227 IDSxpx86!ccLib::CThread::ThreadProcStatic+0x80                 
12 000000000568fcb0 000000006e566d80 IDSxpx86!ccLib::CThread::ThreadProcCRT+0xc                     
13 (Inline)         ---------------- IDSxpx86!invoke_thread_procedure+0xd                           
14 000000000568fcbc 00000000758962c4 IDSxpx86!thread_start<unsigned int (__stdcall*)(void *)>+0x58  
15 000000000568fcf8 00000000773a1b69 KERNEL32!BaseThreadInitThunk+0x24                              
16 000000000568fd0c 00000000773a1b34 ntdll_77340000!__RtlUserThreadStart+0x2f                       
17 000000000568fd54 0000000000000000 ntdll_77340000!_RtlUserThreadStart+0x1b                        

0: kd> dt IDSvia64!g_poDriver m_oIDSEngineManager.m_bNetStatEnabled
0xffff9588`47404000 
   +0x18e0 m_oIDSEngineManager                   : 
      +0x154 m_bNetStatEnabled                     : 0n1

0: kd> dt IDSvia64!g_poDriver m_oMessageChannelManager.m_oMessageQueue.m_dwQueueSize
0xffff9588`47404000 
   +0xf80 m_oMessageChannelManager                               : 
      +0x008 m_oMessageQueue                                        : 
         +0x000 m_dwQueueSize                                          : 0x26d0

Environment

SEP 14.3x, SEDR

Resolution

To resolve the issue, disable the below:

1. Disable IPS submissions in SEPM
2. Disable Netstat in SEDR

IPS submissions can be disabled in SEPM by unchecking the below:
a. Navigate to SEPM > Clients > Policies > Click on External Communications link.
b. In Submissions tab click on "More options..." button
c. Uncheck "Network data that helps Symantec recommend reductions to your organization's network attack surface"

How to disable Netstat is SEDR:
a. Login to SEDR web UI, go to Settings > Global > select SEPM Connector you need to modify
b. Choose 'Endpoint Activity Recorder Configuration' from menu


c. Uncheck the 'Enable Netstat Event Recording' checkbox, then save the change.
d. New FDR policy will be propagated to SEPM and then SEP will pick up

 

 

Attachments