Endpoint Protection client fails to update content until SEP service is restarted

Products

• Endpoint Protection

Issue/Introduction

Symantec Endpoint Protection (SEP) client fails to update content until the Symantec Endpoint Protection service is restarted or the device reboots.  When this issue occurs, the SEP client will also remain disconnected from the Symantec Endpoint Protection Manager (SEPM), meaning no heartbeat can occur, which thus prevents the client from receiving new policy updates or commands from the Symantec Endpoint Protection Manager.

Checking the cve.log will confirm whether SEP client/Manager communication has ceased.  While in this state, scanning will still occur, but communication to the manager as well as updates will fail.

SEP System logs:

Error An update for Virus and Spyware Definitions SDS Win64 failed to install. Error: Content patching failure (0xE0010001), DuResult: Catalog callback failed (60).

Error An update for SONAR Definitions failed to install.  Error: Content patching failure (0xE0010005), DuResult: Success (0).

Performance Monitor:

Virtual Bytes of one of ccSvcHst.exe is nearly 2 GB.

Cause

The issue is a result of ccSvcHst.exe heap fragmentation issue that occurs when multiple sets of definitions are mapped into the process simultaneously and the heap exceeds a 32-bit user space memory limit and crashes. (e.g. when a new set of definitions are downloaded).

In Windows, a reserved area of memory is created for each process that is started. This memory area is called the heap, because it consists of contiguous (i.e. heaped together) memory pages. In addition to that default heap, a process can create a private heap that consists of blocks of memory in its private address space. These blocks of memory get filled up with both small and large memory allocations. Heap fragmentation arises when e.g. larger allocations are freed in a block, but the smaller ones stick around. The fragmentation makes it impossible for the heap manager to perform the necessary cleanup and eventually leads to a point of failure.

Environment

Symantec Endpoint Protection 14.x.

Resolution

14.2 client workaround options:

• For clients still running version 14.2.x you can manually enable a new code feature to monitor and flush the memory space automatically.
• For 14.3.x clients see below in 'History' as these settings are not required.  

Symantec Endpoint Protection 14.2 MP1 partially addresses this issue by actively monitoring and resolving memory fragmentation within ccSvcHst.exe as it occurs. This memory management feature is optional and is only enabled by creating the following registry value(s) in 14.2 MP1 and later only:

Note:  Tamper Protection needs to be temporarily set to Log only to make this registry change.  Revert the Tamper Protection settings to their previous configuration afterwards.

(32-bit machines)
HKLM\Software\Symantec\Symantec Endpoint Protection\SMC

(64-bit machines)
HKLM\SOFTWARE\WOW6432Node\Symantec\Symantec Endpoint Protection\SMC

1. MemoryMonitor (DWORD) - Amount of memory that needs to be available.
   • Valid (Value data) are 350 to 500 MB (decimal). This value is required. Valid values will enable the feature. 
   • Invalid values will enable the feature with a default of 350 MB.
   • A missing registry value or the value set to 0 will disable the feature. 
      
2. MemoryMonitorFreq (DWORD) - Time in between checks.  Valid (Value data) are 1 to 24 hours (decimal). Default value of 8 hours is taken when this value is not created. (Optional)

Settings are only checked once at start. To change settings, smc restart or system reboot is required.

Note: This value should be set only on machines experiencing the related symptoms.  Typical impacted systems include Citrix hosts, Hyper-V workstations, systems with high numbers of terminal services users logged into them. 

History

An intermediate fix was provided on November 13, 2017, in the form of SDS 1.5.0.321, which reduced memory reservation in certain scenarios by ~50%. ccSvcHst.exe related memory usage was further improved in SEP 14 RU1 MP1

14.3+ tuned the memory monitor using data from 14.2 to better anticipate and handle the memory overflow and recycle the memory space more effectively.  A not yet released future build of SEP 14 will fully address the issue at the core and the memory overflow will no longer occur.

Workarounds

The following will help reduce the probability of encountering the symptoms related to this issue as well:

• Moving from Dark Network Definitions to Standard Definitions,  Dark network in memory memory requirements are 2-3x as much,  moving to Standard definitions sets if possible will prevent most instances of the crash.
   
• Restarting SMC at regular intervals to allow ccSvcHst.exe to start afresh with a new heap. This must be scheduled manually with smc stop/start commands. Note that this workaround is unnecessary if using SEP 14.2 MP1 with MemoryMonitor enabled as described above.

Note on the SMC stop/start command: In some environments where you have not disabled WSC (Windows Security Center) notifications, you may see Windows Security Center popup stating: "Windows Firewall and Symantec Endpoint Protection are both turned off. Tap or click to see available options." This pop up is expected when the SMC is stopped and can be suppressed as below:

Method 1 – Manually

1. Click on the start button, then click on ‘Settings’ > Click on ‘System’.

2. Click on ‘Notifications & actions’ > Disable ‘Security and Maintenance’ notifications or disable “Show Notification banners”

 

Method 2 – By Registry Key

1. To disable notifications create the DWORD key “Enabled” and set the value to 0.

2. To enable notifications remove the DWORD key “Enabled” completely.

Registry Key:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Notifications\Settings\Windows.SystemToast.SecurityAndMaintenance

Value:

“Enabled”=dword:00000000

 

Method 3 – By GPO

1. Navigate to Computer Configuration, Policies, Windows Settings, Security Settings, Windows Firewall with Advanced Security, Windows Firewall with Advanced Security: Windows Firewall Properties.

2. Here you can configure the notifications for each profile using Settings, Customize, Firewall settings, Display a notification.

Additional Information

SepMasterService cannot start after service restart by MemoryMonitor function