Symantec Endpoint Protection (SEP) client fails to update content until the Symantec Endpoint Protection service is restarted.

SEP System logs:

Error An update for Virus and Spyware Definitions SDS Win64 failed to install. Error: Content patching failure (0xE0010001), DuResult: Catalog callback failed (60).

Error An update for SONAR Definitions failed to install.  Error: Content patching failure (0xE0010005), DuResult: Success (0).

Performance Monitor:

Virtual Bytes of one of ccSvcHst.exe is nearly 2 GB.

The issue is a result of ccSvcHst.exe heap fragmentation that occurs when multiple sets of definitions are mapped into the process simultaneously (e.g. when a new set of definitions are downloaded).

In Windows, a reserved area of memory is created for each process that is started. This memory area is called the heap, because it consists of contiguous (i.e. heaped together) memory pages. In addition to that default heap, a process can create a private heap that consists of blocks of memory in its private address space. These blocks of memory get filled up with both small and large memory allocations. Heap fragmentation arises when e.g. larger allocations are freed in a block, but the smaller ones stick around. The fragmentation makes it impossible for the heap manager to perform the necessary cleanup and eventually leads to a point of failure.

Symantec Endpoint Protection 14.2 MP1 addresses this issue by actively monitoring and resolving memory fragmentation within ccSvcHst.exe as it occurs. This memory management is optional and is enabled by creating the following registry value(s) in 14.2 MP1 and later only:

Note:  Tamper Protection needs to be temporarily set to Log only to make this registry change.  Revert the Tamper Protection settings to their previous configuration afterwards.

(32-bit machines)
HKLM\Software\Symantec\Symantec Endpoint Protection\SMC

(64-bit machines)
HKLM\SOFTWARE\WOW6432Node\Symantec\Symantec Endpoint Protection\SMC

1. MemoryMonitor (DWORD) - Amount of memory that needs to be available.
   • Valid (Value data) are 350 to 500 MB (decimal). This value is required. Valid values will enable the feature. 
   • Invalid values will enable the feature with a default of 350 MB.
   • A missing registry value or the value set to 0 will disable the feature. 
      
2. MemoryMonitorFreq (DWORD) - Time in between checks.  Valid (Value data) are 1 to 24 hours (decimal). Default value of 8 hours is taken when this value is not created. (Optional)

Settings are only checked once at start. To change settings, smc restart or system reboot is required.

Note: This value should be set only on machines experiencing the related symptoms.

History

An intermediate fix was provided on November 13, 2017, in the form of SDS 1.5.0.321, which reduced memory reservation in certain scenarios by ~50%. ccSvcHst.exe related memory usage was further improved in SEP 14 RU1 MP1

Workarounds

The following will help reduce the probability of encountering the symptoms related to this issue:

• Moving from Dark Network Definitions to Standard Definitions.
   
• Restarting SMC at regular intervals to allow ccSvcHst.exe to start afresh with a new heap. This must be scheduled manually with smc stop/start commands. Note that this workaround is unnecessary if using SEP 14.2 MP1 with MemoryMonitor enabled as described above.

Note: In some environments, you may see Windows Security Center popup stating: "Windows Firewall and Symantec Endpoint Protection are both turned off. Tap or click to see available options." This pop up is expected when the SMC is stopped and can be suppressed as below: