High CPU usage linked to ccSvcHst.exe process/service

Products

• Endpoint Protection

Issue/Introduction

A system with Symantec Endpoint Protection (SEP) is experiencing high CPU usage due to ccSvcHst.exe. It is possible to reboot the system (see the Related Articles section if that should not be the case).

Environment

SEP 12.1, 14 or higher

Resolution

Follow these steps in this order.

Generate a ccSvcHst.exe process dump

1. Download ProcDump.
2. Right-click Procdump.zip, select Extract All... and extract the files to the Windows folder.
3. Open a Command Prompt (cmd.exe) window.
4. Run the command procdump –ma -c <CPU usage percentage that will trigger a dump> <Process ID of high CPU ccsvchst.exe process> ccsvchst.dmp (e.g. run the command procdump -ma -c 75 2300 ccsvchst.dmp to generate a dump when the CPU usage for the ccSvcHst.exe with process ID 2300 is at least 75%).

Note: The process ID of the offending ccSvcHst.exe process can be determined in the following way:

1. Right-click the Windows task bar and select Start Task Manager.
2. Navigate to the Processes tab and click the CPU column header button to sort the processes by CPU usage.
3. Make note of the offending ccSvcHst.exe process' CPU usage. If the PID column is not visible, navigate to View > Select Columns, tick PID (Process Identifier), then click the OK button.

Generate a Windows Performance Recorder trace file

1. Download and install the Windows Performance Toolkit.
2. Run Windows Performance Recorder. Set the following options, then click the Start button to capture the issue:

   a. Under Select additional profiles for performance recording, under Resource Analysis, select CPU Usage, Disk I/O Activity and File I/O Activity. Under Scenario Analysis, tick Minifilter I/O activity.
   b. Performance scenario: General.
   c. Detail level: Verbose.
   d. Logging mode: File.

3. After reproducing the issue, click the Save button, browse to the location where you wish to save the trace file and click the Save button. 
4. When saved, click the Open Folder button to navigate to the save location, select all files, right-click on them and select the Send to > Compressed (zipped) folder menu option.

Generate a low-altitude Process Monitor trace

1. Download : See attachements below
2. Right-click Procmon24Low.zip, select Extract All..., and extract it to a location of your choice.
3. Navigate to that location, then run Procmon24Low.exe.
4. When the Process Monitor Filter pop-up window is shown, click on the Reset button, then Apply and OK.
5. In the File menu, press Ctrl-E, then Ctrl-X to stop the capture and clear the display.
6. In the Filter menu, ensure Enable Advanced Output is ticked.
7. Press Ctrl-E to start capturing.
8. Capture the issue for a minute or two, return to the Procmon window and press Ctrl-E to stop capturing.
9. Press Ctrl-S and save all events in the Native Process Monitor Format (PML).
10. When saved, navigate to the save location, select and right-click the PML file, then select the Send to > Compressed (zipped) folder menu option to compress it.

Generate a complete memory dump

• If the system is a virtualized one:

  • VMware: generate a snapshot, then follow the steps described in https://kb.vmware.com/kb/2003941 ("Converting a snapshot file to memory dump using the vmss2core tool") to convert it to a memory dump.
  • Hyper-V: generate a snapshot using LiveKD, as described in https://blogs.msdn.microsoft.com/vimalsdesk/2014/11/23/taking-a-dump-of-a-vm-running-on-hyper-v/
  • Citrix: follow the steps outlined in https://support.citrix.com/article/CTX123177 ("How to Trigger a Memory Dump from a Windows Virtual Machine Running on XenServer")

• If the system is a physical one:

  1. Open Registry Editor (regedit.exe).
  2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\.
  3. Double-click CrashDumpEnabled, change the value to 1 (1 = complete dump, 2 = kernel dump) and click OK.
  4. Close Registry Editor.
  5. Click the Start button, right-click Computer and select Properties. Click Advanced System Settings.
  6. In the Performance area, click the Settings... button.
  7. In the Performance Options window, navigate to the Advanced tab, then click the Change... button.
  8. Click the Custom size radio button, then set both Initial size (MB) and Maximum size (MB) to at least the amount of system memory + 257 MB, by entering the correct value in each field and clicking the "Set" button when done. E.g. if the system has 4 GB of memory, set both fields to (4 x 1024) + 257 = 4353 MB. If the system has 8 GB of memory, set both fields to (8 x 1024) + 257 = 8449 MB.
  9. After having made these changes, restart the system.
  10. Download https://download.sysinternals.com/files/NotMyFault.zip and unpack the archive to C:\Windows. Open a Command Prompt (cmd.exe) window and, without pressing Enter at the end, type in the command notmyfault /accepteula /crash. Reproduce the issue, return to the Command Prompt window and press Enter to forcefully crash the system.

Upload the resulting dump and all other data to an existing case (or create a new one) using SymDiag

1. Download and run SymDiag
2. Click Collect Data for Support.
3. In the Select Products section, tick Endpoint Protection Client and click Next.
4. In the Select Data Type section, under Data Type, select All data, tick Choose additional files to collect and click Next.
5. Below Choose additional files to collect, click the Browse... button, navigate to and select the dump created above (typically C:\Windows\MEMORY.DMP) and all other generated data, then click the Open button, followed by the Next button.
6. After the data collection has finished, enter your name, company, case number, contact information and a brief description of the issue and click the Open or Update a Support Case button. Enter user name and password, then click the Login button.