Endpoint protection clients generating huge traffic on manager

Products

• Endpoint Protection

Issue/Introduction

Endpoint protection (SEP) clients generating huge traffic on SEP manager (SEPM) communication port.

Access log shows HTTP response 503 indicating load on SEPM

exsecars log shows : GetIndex 412 Register again

10/11 00:10:36 [4620:4364] SendIndexFileToClient: <SEP Client IP Address> GetIndex 412 Register again

ersecreg -a.log

10/11 00:10:36 [4620:4364] ##.##.##.##<AgentInfo PreferredMode="1" DomainID="#################" AgentType="105" AgentID="###############" HardwareKey="###############" UserDomain="LocalComputer" LoginUser="#####" ComputerDomain="example.com" ComputerName="Example-Computer" PreferredGroup="My%20Company%5CWindows_Server%5####" SiteDomainName="" AgentPlatform="Windows%20Server%202016" IsNPVDIClient="0"/> AgentID=############### AgentType=105 ComputerID=############### Hash Key=###############

AgentRegister-1.log

2023-10-11 00:10:36.451 THREAD 78 WARNING: AgentRegisterHandler> registerClient>> hardwareId: ###############
2023-10-11 00:10:36.469 THREAD 78 WARNING: AgentRegisterHandler> registerClient>> Time checkDomainIdExistence: ###########
2023-10-11 00:10:36.470 THREAD 78 WARNING: AgentRequestHandler> registerClientMain>> Begin... client: Computer Example-Computer
2023-10-11 00:10:36.473 THREAD 78 WARNING: AgentRequestHandler> registerClientMain>> Done! (exactly matched in computer mode - update) ClientId: ##################; old client got deleted, OldClientId: ###############
2023-10-11 00:10:36.473 THREAD 78 WARNING: AgentRegisterHandler> registerClient>> Query Done! clientId: ###################
2023-10-11 00:10:36.473 THREAD 78 WARNING: AgentRegisterHandler> registerComputer>> ComputerId: #####################
2023-10-11 00:10:36.481 THREAD 78 WARNING: BatchUpdater>>flushing...: SEM_COMPUTER_NIC UPDATE
2023-10-11 00:10:36.509 THREAD 78 WARNING: BatchUpdater>>flushing...: SEM_COMPUTER_NIC UPDATE
2023-10-11 00:10:36.524 THREAD 78 WARNING: AgentRegisterHandler> registerComputer>> Query Done! foundExisting: true, computerId: ##################
2023-10-11 00:10:36.524 THREAD 78 WARNING: AgentRegisterHandler> registerAgent>> Registering agent..., agentType: 105
2023-10-11 00:10:36.532 THREAD 78 WARNING: AgentRegisterHandler> registerAgent>> Found existing agent. agentId: ####################
2023-10-11 00:10:36.533 THREAD 78 WARNING: AgentRegisterHandler> registerAgent>>  Query Done! foundExisting: true, agentId: ####################
2023-10-11 00:10:36.566 THREAD 78 WARNING: AgentRegisterHandler> agentRegister>> Update client, ClientId: #######################
2023-10-11 00:10:36.571 THREAD 78 WARNING: AgentRegisterHandler> agentRegister>> Update computer, computerId: #################
2023-10-11 00:10:36.575 THREAD 78 WARNING: AgentRegisterHandler> agentRegister>> Update agent, AgentId: #######################
2023-10-11 00:10:36.580 THREAD 78 WARNING: AgentRegisterHandler> agentRegister>> newclient: false
2023-10-11 00:10:36.618 THREAD 78 WARNING: AgentRegisterHandler> agentRegister>> Committing db changes for clientId: ####################

scm-server-0.log shows SEPM experienced deadlock resulting in secars connection failure

2023-10-11 11:32:04.965 THREAD 1 SEVERE: ================== Server Environment ===================
2023-10-11 13:21:09.263 THREAD 35 SEVERE: Unexpected server error. in: com.sygate.scm.server.task.PackageTask
com.sygate.scm.server.metadata.MetadataException: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
2023-10-11 13:21:30.548 THREAD 40 SEVERE: Unexpected server error. in: com.sygate.scm.server.task.IISCacheTask
com.sygate.scm.server.metadata.MetadataException: Transaction (Process ID 159) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
2023-10-11 16:53:08.042 THREAD 40 SEVERE: Error in ClientTransportInfoTask starting client transport after SEPM service started
java.net.ConnectException: Connection refused: connect
2023-10-11 17:19:06.884 THREAD 5025 SEVERE: Error in ClientTransportInfoTask starting client transport after SEPM service started
java.net.ConnectException: Connection refused: connect
// below is the last line in scm-server-0.log
2023-10-11 17:19:51.377 THREAD 5217 WARNING: Log table switched to: SERVER_CLIENT_LOG_2, old table estimated row count: 11012, to add row count in new table: 994, last switch time: 2023-10-11 17:19:00 

scm-server-0.log also shows "re-register":

2023-10-10 22:40:41.638 THREAD 40 WARNING: Found agent ################## is trying to update its status to SEPM, but it is not existing in database, will force it re-register
2023-10-10 22:52:41.650 THREAD 40 WARNING: Found agent ################## is trying to update its status to SEPM, but it is not existing in database, will force it re-register
2023-10-10 22:59:41.658 THREAD 40 WARNING: Found agent ################## is trying to update its status to SEPM, but it is not existing in database, will force it re-register
2023-10-10 22:59:41.659 THREAD 40 WARNING: Found agent ################## is trying to update its status to SEPM, but it is not existing in database, will force it re-register

After SEPM gets the clients' online/offline state from Secars, it tries to update its DB. It then finds many clients in the Secars response didn't exist in SEPM DB. This caused SEPM response to those clients for a re-register.

 

Resolution

Stop SEPM services and perform the steps below.

• Edit httpd.conf:

Modify Listen :8014 To Listen *:8014

Note: Make changes in all SEPMs, one at a time and then restart services. Keep a gap of 10 mins before editing next server's httpd.conf.

• In conf.properties add following line to stop re-registration:

scm.agent.onlineStatus.reset=0

• Enable the SEPM database transaction isolation level READ_COMMITTED_SNAPSHOT
  Refer KB

After it is turned on, add the following config in the conf.properties file to prevent the SEPM upgrade from turning it off:

scm.upgrade.disable.database.snapshot=false