Endpoint Protection client fails to update content until SEP service is restarted
Article ID: 170255
Updated On:
Products
Issue/Introduction
Symantec Endpoint Protection (SEP) client fails to update content until the Symantec Endpoint Protection service is restarted or the device reboots. When this issue occurs, the SEP client may also remain disconnected from the Symantec Endpoint Protection Manager (SEPM), meaning no heartbeat can occur, which thus prevents the client from receiving new policy updates or commands from the Symantec Endpoint Protection Manager.
Checking the cve.log will confirm whether SEP client/Manager communication has ceased. While in this state, scanning will still occur, but communication to the manager as well as updates will fail.
SEP System logs:
Error An update for Virus and Spyware Definitions SDS Win64 failed to install. Error: Content patching failure (0xE0010001), DuResult: Catalog callback failed (60).
Error An update for SONAR Definitions failed to install. Error: Content patching failure (0xE0010005), DuResult: Success (0).
Performance Monitor:
Virtual Bytes of one of ccSvcHst.exe is nearly 2 GB.
Environment
Symantec Endpoint Protection 14.x.
Cause
The issue is a result of ccSvcHst.exe heap fragmentation issue that occurs when multiple sets of definitions are mapped into the process simultaneously and the heap exceeds a 32-bit user space memory limit and crashes. (e.g. when a new set of definitions are downloaded).
In Windows, a reserved area of memory is created for each process that is started. This memory area is called the heap, because it consists of contiguous (i.e. heaped together) memory pages. In addition to that default heap, a process can create a private heap that consists of blocks of memory in its private address space. These blocks of memory get filled up with both small and large memory allocations. Heap fragmentation arises when e.g. larger allocations are freed in a block, but the smaller ones stick around. The fragmentation makes it impossible for the heap manager to perform the necessary cleanup and eventually leads to a point of failure.
Resolution
This issue is fixed in Symantec Endpoint Protection 14.3 RU5 for 64-bit endpoints. For information on how to obtain the latest build of Symantec Endpoint Protection, see Download the latest version of Symantec software here.
Note After upgrading to 14.3 RU5 you can safely remove MemoryMonitor and MemoryMonitorFreq described below from the registry.
Option for older SEP clients:
For clients still running version 14.2.x (or 14.3.x older than 14.3.8259 RU5) you can manually enable a new code feature to monitor and flush the memory space automatically. This memory management feature is optional and is only enabled by creating the following registry value(s) in 14.2 MP1 and later:
Note: Tamper Protection needs to be temporarily set to Log only to make this registry change. Revert the Tamper Protection settings to their previous configuration afterwards.
(32-bit machines)
HKLM\Software\Symantec\Symantec Endpoint Protection\SMC
(64-bit machines)
HKLM\SOFTWARE\WOW6432Node\Symantec\Symantec Endpoint Protection\SMC
- MemoryMonitor (DWORD) - Amount of memory that needs to be available.
- Valid (Value data) are 350 to 500 MB (decimal). This value is required. Valid values will enable the feature.
- Invalid values will enable the feature with a default of 350 MB.
- A missing registry value or the value set to 0 will disable the feature.
- MemoryMonitorFreq (DWORD) - Time in between checks. Valid (Value data) are 1 to 24 hours (decimal). Default value of 8 hours is taken when this value is not created. (Optional)
Settings are only checked once at start. To change settings, smc restart or system reboot is required.
Note: This value should be set only on machines experiencing the related symptoms. Typical impacted systems include Citrix hosts, Hyper-V workstations, systems with high numbers of terminal services users logged into them.
History
An intermediate fix was provided on November 13, 2017, in the form of SDS 1.5.0.321, which reduced memory reservation in certain scenarios by ~50%. ccSvcHst.exe related memory usage was further improved in SEP 14 RU1 MP1
14.3+ tuned the memory monitor using data from 14.2 to better anticipate and handle the memory overflow and recycle the memory space more effectively.
Fix is included in version 14.3 RU5 x64 Agent
Workarounds
The following will help reduce the probability of encountering the symptoms related to this issue as well:
- Moving from Dark Network Definitions to Standard Definitions, Dark network in memory requirements are 2-3x as much, moving to Standard definitions sets if possible will prevent most instances of the crash.
- Restarting SMC at regular intervals to allow ccSvcHst.exe to start afresh with a new heap. This must be scheduled manually with smc stop/start commands. Note that this workaround is unnecessary if using SEP 14.2 MP1 with MemoryMonitor enabled as described above.
Note on the SMC stop/start command: In some environments where you have not disabled WSC (Windows Security Center) notifications, you may see Windows Security Center popup stating: "Windows Firewall and Symantec Endpoint Protection are both turned off. Tap or click to see available options." This pop up is expected when the SMC is stopped and can be suppressed as below:
Additional Information
Feedback
