Endpoint searches hang

Products

• Endpoint Detection and Response

Issue/Introduction

Endpoint searches appear to be stopped or hung in the SEDR web interface. They do not appear to progress when you monitor them.

The search does not progress

Environment

This will not affect the S550 or 8880 (dell R730) where the SEDR appliance has more than 192 GB of memory installed.  All other appliances could experience this.

Cause

There is a CPU usage spike.  This could manifest in EDR by showing different symptoms.

• In the epmp_r3.log file Excessive "TokenException" warnings start showing.  Heap starvation may be occurring and some processes may get blocked indefinitely.

Example:

2021-01-23 05:05:46,587 [http-nio-127.0.0.1-8011-exec-15] WARN c.s.p.identity.tokens.AmqpConnection - AMQP Connection to RabbitMQ [[localhost:5672]:/] successful!

2021-01-27 06:01:06,071 [http-nio-127.0.0.1-8011-exec-185] WARN com.symantec.platform.r3.router.R3Request - unexpected error parsing the token com.symantec.platform.identity.tokens.TokenException: invalid_token - General token decode failure: Invalid serialized unsecured/JWS/JWE object: Missing part delimiters

• epmp_r3 shows timeout.  Other services may also show they are timing out.  There is resource starvation occurring which could result in swaps occurring for partition access.

Resolution

This is resolved in SEDR 4.6 please upgrade to EDR version 4.6.

Additional Information

• This could be an indication that the environment is overused and there are too many endpoints for a virtual EDR appliance.  If there are more than 10,000 endpoints configured in the customer environment this does not meet the information reported in the current Sizing recommendations for the virtual appliance and what is stated in the Symantec EDR platform support matrix technical documents for SEDR 4.6.
• TSE see internal notes.