High CPU usage linked to ccSvcHst.exe process/service

Products

• Endpoint Protection

Issue/Introduction

A system with Symantec Endpoint Protection (SEP) is experiencing high CPU or Memory usage due to ccSvcHst.exe.

Environment

• SEP 14.3.x
• SEP 16

Resolution

Gathering a Windows Performance Recorder trace and SymDiag while reproducing the issue is typically sufficient when investigating potential performance issues involving SEP.   While there are other pieces of data mentioned below, a WPR and SymDiag are generally enough to investigate a performance issue.

Follow these steps below to generate a WPR trace file while reproducing the issue. 

Gather this data first

Steps to gather a Windows Performance Recorder Trace and SymDiag

1. Download and install the Windows Performance Recording
   1. Download Windows Assessment and Deployment Kit.
   2. When selecting which features to install only choose Windows Performance Recorder.
2. Run Windows Performance Recorder. Click More Options
3. Set the following options, then click the Start button to capture the issue:

   a. Under Select additional profiles for performance recording, under Resource Analysis, select CPU Usage, Disk I/O Activity and File I/O Activity. Under Scenario Analysis, tick Minifilter I/O activity.
   b. Performance scenario: General.
   c. Detail level: Verbose.
   d. Logging mode: File.

4. Click Start and then reproduce the issue.
5. Note the timestamp the issue was reproduced and any other supporting evidence demonstrating SEP is causing an issue.
6. After reproducing the issue, click the Save button, browse to the location where you wish to save the trace file and click the Save button. 
7. Once Saved, exit WPR and locate the saved Windows Performance Analyzer Trace file (.etl) and compress (zip) the file.

A SymDiag with All Data should also be gathered after reproducing the issue.

1. Download and run SymDiag
2. Click Collect Data for Support.
3. In the Select Products section, tick Endpoint Protection Client and click Next.
4. In the Select Data Type section, under Data Type, select All data, tick Choose additional files to collect and click Next.
5. After the data collection has finished, save the file and upload it to the case.

 

If necessary and instructed to by Support

Only gather one or more of these pieces of data below if instructed by support.  

Generate a ccSvcHst.exe process dump

1. Download ProcDump.
2. Right-click Procdump.zip, select Extract All... and extract the files to the Windows folder.
3. Open a Command Prompt (cmd.exe) window.
4. Run the command procdump –ma -c <CPU usage percentage that will trigger a dump> <Process ID of high CPU ccsvchst.exe process> ccsvchst.dmp (e.g. run the command procdump -ma -c 75 2300 ccsvchst.dmp to generate a dump when the CPU usage for the ccSvcHst.exe with process ID 2300 is at least 75%).

Note: The process ID of the offending ccSvcHst.exe process can be determined in the following way:

1. Right-click the Windows task bar and select Start Task Manager.
2. Navigate to the Processes tab and click the CPU column header button to sort the processes by CPU usage.
3. Make note of the offending ccSvcHst.exe process' CPU usage. If the PID column is not visible, navigate to View > Select Columns, tick PID (Process Identifier), then click the OK button.

 

Generate a low-altitude Process Monitor trace

1. 1. Download Process Monitor > https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
   2. Unzip the downloaded file.
   3. Open an administrative command prompt and navigate to the location of the unzipped files.
   4. Verify that PROCMON24 is not already running the following command.
      

       fltmc 

      
      Note: If PROCMON24 is already running, then reboot the machine (please see the example below where PROCMON24 is already running)
      

       C:\ProcessMonitor>fltmc
      
       Filter Name                     Num Instances    Altitude    Frame
       ------------------------------  -------------  ------------  -----
       bindflt                                 1       409800         0
       UCPD                                    3       385250.5       0
       BHDrvx64                                3       365100         0
       SymEvnt                                 3       365090         0
       eeCtrl                                  2       329010         0
       SRTSP                                   3       329000         0
       vsepflt                                 3       328200         0
       SymEFASI                                3       260610         0
       storqosflt                              0       244000         0
       wcifs                                   0       189900         0
       CldFlt                                  0       180451         0
       bfs                                     5       150000         0
       FileCrypt                               0       141100         0
       luafv                                   1       135000         0
       UnionFS                                 0       130850         0
       npsvctrig                               1        46000         0
       Wof                                     1        40700         0
       FileInfo                                3        40500         0
       PROCMON24                               0        40000         0
      

   5. From the administrative command line, start procmon using the altitude flag.

   Note: The value used should be lower than item in the filter stack that is to be monitored.

   procmon.exe /Altitude 40000

   6. When the Process Monitor Filter pop-up window is shown, click the Reset button, then Apply and OK.
   7. In the File menu, press Ctrl-E, then Ctrl-X to stop the capture and clear the display.
   8. In the Filter menu, ensure "Enable Advanced Output" is checked.
   9. Press Ctrl-E to start capturing.
   10. Capture the issue for a minute or two, return to the Procmon window and press Ctrl-E to stop capturing.
   11. Press Ctrl-S and save all events in the Native Process Monitor Format (PML).
   12. When saved, navigate to the save location, select and right-click the PML file, then select the Send to > Compressed (zipped) folder menu option to compress it.
    

Generate a complete memory dump

• If the system is a virtualized one:

  • VMware: generate a snapshot, then follow the steps described in https://kb.vmware.com/kb/2003941 ("Converting a snapshot file to memory dump using the vmss2core tool") to convert it to a memory dump.
  • Hyper-V: generate a snapshot using LiveKD, as described in LiveKd - Sysinternals | Microsoft Learn
  • Citrix: follow the steps outlined in https://support.citrix.com/article/CTX123177 ("How to Trigger a Memory Dump from a Windows Virtual Machine Running on XenServer")

• If the system is a physical one:

  1. Open Registry Editor (regedit.exe).
  2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\.
  3. Double-click CrashDumpEnabled, change the value to 1 (1 = complete dump, 2 = kernel dump) and click OK.
  4. Close Registry Editor.
  5. Click the Start button, right-click Computer and select Properties. Click Advanced System Settings.
  6. In the Performance area, click the Settings... button.
  7. In the Performance Options window, navigate to the Advanced tab, then click the Change... button.
  8. Click the Custom size radio button, then set both Initial size (MB) and Maximum size (MB) to at least the amount of system memory + 257 MB, by entering the correct value in each field and clicking the "Set" button when done. E.g. if the system has 4 GB of memory, set both fields to (4 x 1024) + 257 = 4353 MB. If the system has 8 GB of memory, set both fields to (8 x 1024) + 257 = 8449 MB.
  9. After having made these changes, restart the system.
  10. Download https://download.sysinternals.com/files/NotMyFault.zip and unpack the archive to C:\Windows. Open a Command Prompt (cmd.exe) window and, without pressing Enter at the end, type in the command notmyfault /accepteula /crash. Reproduce the issue, return to the Command Prompt window and press Enter to forcefully crash the system.