Symantec Endpoint Protection (SEP) client fails to update content until the Symantec Endpoint Protection service is restarted or the device reboots. When this issue occurs, the SEP client may also remain disconnected from the Symantec Endpoint Protection Manager (SEPM), meaning no heartbeat can occur, which thus prevents the client from receiving new policy updates or commands from the Symantec Endpoint Protection Manager.
Checking the cve.log will confirm whether SEP client/Manager communication has ceased. While in this state, scanning will still occur, but communication to the manager as well as updates will fail.
Error An update for Virus and Spyware Definitions SDS Win64 failed to install. Error: Content patching failure (0xE0010001), DuResult: Catalog callback failed (60).
Error An update for SONAR Definitions failed to install. Error: Content patching failure (0xE0010005), DuResult: Success (0).
Virtual Bytes of one of ccSvcHst.exe is nearly 2 GB.
Symantec Endpoint Protection 14.x.
The issue is a result of ccSvcHst.exe heap fragmentation issue that occurs when multiple sets of definitions are mapped into the process simultaneously and the heap exceeds a 32-bit user space memory limit and crashes. (e.g. when a new set of definitions are downloaded).
In Windows, a reserved area of memory is created for each process that is started. This memory area is called the heap, because it consists of contiguous (i.e. heaped together) memory pages. In addition to that default heap, a process can create a private heap that consists of blocks of memory in its private address space. These blocks of memory get filled up with both small and large memory allocations. Heap fragmentation arises when e.g. larger allocations are freed in a block, but the smaller ones stick around. The fragmentation makes it impossible for the heap manager to perform the necessary cleanup and eventually leads to a point of failure.
This issue is fixed in Symantec Endpoint Protection 14.3 RU5 for 64-bit endpoints. For information on how to obtain the latest build of Symantec Endpoint Protection, see Download the latest version of Symantec software here.
Note After upgrading to 14.3 RU5 you can safely remove MemoryMonitor and MemoryMonitorFreq described below from the registry.
Option for older SEP clients:
For clients still running version 14.2.x (or 14.3.x older than 14.3.8259 RU5) you can manually enable a new code feature to monitor and flush the memory space automatically. This memory management feature is optional and is only enabled by creating the following registry value(s) in 14.2 MP1 and later:
Note: Tamper Protection needs to be temporarily set to Log only to make this registry change. Revert the Tamper Protection settings to their previous configuration afterwards.
HKLM\Software\Symantec\Symantec Endpoint Protection\SMC
HKLM\SOFTWARE\WOW6432Node\Symantec\Symantec Endpoint Protection\SMC
Settings are only checked once at start. To change settings, smc restart or system reboot is required.
Note: This value should be set only on machines experiencing the related symptoms. Typical impacted systems include Citrix hosts, Hyper-V workstations, systems with high numbers of terminal services users logged into them.
An intermediate fix was provided on November 13, 2017, in the form of SDS 220.127.116.111, which reduced memory reservation in certain scenarios by ~50%. ccSvcHst.exe related memory usage was further improved in SEP 14 RU1 MP1
14.3+ tuned the memory monitor using data from 14.2 to better anticipate and handle the memory overflow and recycle the memory space more effectively.
Fix is included in version 14.3 RU5 x64 Agent
The following will help reduce the probability of encountering the symptoms related to this issue as well:
Note on the SMC stop/start command: In some environments where you have not disabled WSC (Windows Security Center) notifications, you may see Windows Security Center popup stating: "Windows Firewall and Symantec Endpoint Protection are both turned off. Tap or click to see available options." This pop up is expected when the SMC is stopped and can be suppressed as below: