How to manually monitor the number of ActiveThreads on a SPE server for C-Mode NetApp Filers

Products

Protection Engine for NAS

Issue/Introduction

You are about to turn on VSCAN for one or more network shares hosted by NetApp Filers. You seek a way to monitor SPE to make sure it is scanning normally and not exhausting its pool of scanning threads or queue of scan requests waiting for an Active Thread.

Resolution

To monitor ActiveThreads within today's .rcl file

At the cmd prompt, navigate to the folder containing the SPE logs by typing:
```
cd C:\Program Files\Symantec\Scan Engine
```
At the cmd prompt, type the following:
```
findstr "127.0.0.1-[2-4][0-9]-" SSE20231220.rcl | find /c "-"
```
...where 20231220 is today's date in YYYYMMDD format. Note that 127.0.0.1-[2-4][0-9] finds any lines where the connected RPC client is the ONTAP AV Connector installed on the same Windows machine as SPE and the number of Active Threads is anywhere from 20 to 49.
If output is non-zero, or appears to grow when you run the command repeatedly, stop counting and look at the entries. Type:
```
findstr "127.0.0.1-[2-4][0-9]-" SSE20231220.rcl
```
Output will include the following columns separated by the "-" character:
RPC Client-Active Threads-Waiting Threads-Thread Pool size-Threshold for queued requests-Queued Requests-Number of requests(RPC)

Expected output for a medium strength spike is similar to the following (boldface added for emphasis):
1703059065, 6.11667, 3205472423, 353150423692801, Normal, 166857084928, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-23- -48-100-0-2896374, 0.00, 0, , , , , , , , , ,
1703061200, 11.6, 3205514462, 353155668659501, Normal, 166849998848, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-20-28-48-100-0-2914229, 0.00, 0, , , , , , , , , ,
1703080595, 25.6167, 3206278959, 353302104802776, Normal, 166816063488, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-22-26-48-100-0-3043440, 0.00, 0, , , , , , , , , ,
...

...without rising above 3/4 of the maximum number of threads. The boldfaced section is the IP address of the connecting client, then the number of ActiveThreads, followed by the number of WaitingThreads, then the totalThreads in the pool.

Expected output for a SPE failing to catch up to scan requests, similar to the following:
1703115755, 11.6833, 3210218785, 353827700835867, Normal, 166686527488, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-21-27-48-100-0-3438013, 0.00, 0, , , , , , , , , ,
1703115770, 15.9667, 3210226500, 353828085844302, Normal, 166686445568, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-21-27-48-100-0-3438429, 0.00, 0, , , , , , , , , ,
1703115775, 17.9667, 3210228215, 353828134073825, Normal, 166686363648, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-32-16-48-100-0-3438606, 0.00, 0, 10377048064, 11069804544, 4696, 900, 3127.01, 0.01, 9.12, 42555.1, 0, 158965
1703115780, 20.4333, 3210229322, 353828162987498, Normal, 166686343168, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-48-0-48-100-52-3438810, 0.00, 0, , , , , , , , , ,
1703115785, 22.35, 3210231650, 353828232492489, Normal, 166686244864, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-48-0-48-100-99-3438980, 0.00, 0, , , , , , , , , ,
1703115790, 23.6, 3210234432, 353828341468471, Normal, 166685536256, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-48-0-48-100-100-3439109, 0.00, 0, , , , , , , , , ,
1703115795, 23.9833, 3210236027, 353828392558070, Normal, 166685265920, 41013643426, C:\Program Files\Symantec\Scan Engine\log\, 127.0.0.1-48-0-48-100-100-3439184, 0.00, 0, , , , , , , , , ,

Note that in this example, the 48-0-48 pattern shows that all Threads in the pool are Active Threads. The two columns after the TotalThreads are the maximum number of scan requests waiting for an Active Thread, then the current number of scan requests waiting in the queue. Here we see the number of scan requests in the queue does not start rising (100-0, 100-52, 100-99...) until after ActiveThreads is equal to the total Threads in the pool. This is why monitoring the ActiveThreads will give admins a warning before SPE processing becomes unable to recover.

Additional Information

How frequently should I check the ActiveThreads?

When you turn on VSCAN on a share, you should check at least once every ten minutes for the first hour, then hourly. If an hourly check shows a non-zero number of ActiveThreads, you may want to start checking once every ten minutes. If another hour passes and no more growth occurs, or the number of threads goes back down, return to once an hour. You may also check another SPE server to see if the issue is limited to one SPE server, or affects the SPE scanners for a particular NetApp Filer cluster.

What should I do if one SPE server is exhausting its pool of Threads?

Restart the Symantec Protection Engine service to recover resources.

What if all the SPE servers associated with a NetApp Filer cluster exhaust their pool of Threads?

Some admins temporarily turn off their VSCAN to prevent impact to their production network, then check for conflicts with their real-time antivirus scanning or EDR client.