Component Isolation steps for Endpoint Protection and Endpoint Security to assist with troubleshooting
search cancel

Component Isolation steps for Endpoint Protection and Endpoint Security to assist with troubleshooting

book

Article ID: 255167

calendar_today

Updated On:

Products

Endpoint Protection Endpoint Security

Issue/Introduction

This is a troubleshooting guide for Symantec Endpoint Protection (SEP) or Symantec Endpoint Security (SES) agents.  This guide should be used when an issue with the SEP or SES agent is occurring but attempts to find root cause have failed.  Below are steps that can be performed to narrow down which feature, driver, process, policy, rule, setting or other component of the agent may be causing or contributing to the issue.

This method of troubleshooting helps rule out parts of the product by removing or disabling them entirely.  Similar to process of elimination; divide and conquer.  Remove variables from the equation, test again, rinse and repeat until the offending piece is found.  This may seem like a tedious or lengthy process, but it can actually save time when it comes to complex issues especially when there are no error messages present, no logs being written, or other indicators pointing in a specific direction.  Also, throughout this process, when the issue is isolated, this presents a workaround that can be replicated to other affected devices until a permanent fix can be implemented.

These steps may be taken on your own or at the direction of a Support Agent.

Environment

  • Symantec Endpoint Protection (SEP) 14.3.x.
  • Symantec Endpoint Security (SES) 14.3.x.
  • Windows Operating Systems (OS)

Resolution

Note:  The steps below walk through various ways to remove or disable protection features of the SEP/SES agent.  These steps should only be used for troubleshooting or temporary workarounds.  Under normal circumstances we strongly recommend enabling all protection features and policies which allows the agent to provide as much protection as possible.

Table of Contents

About the steps below

Generally these steps are meant to be followed in order, but depending on the specific issue or troubleshooting done so far, you may be able to skip some steps.  They are sorted in magnitude of functional impact to the SEP agent, but don’t necessarily need to be followed chronologically.  Meaning, the first step makes the most changes to the SEP agent as possible, while the last step focuses on fine-tuning policy settings that have a smaller impact.

Notes about the SEP and SES agent

The 14.3.x SEP and SES agent are the same agent, but able to be managed by two different platforms: On-Premise Symantec Endpoint Protection Manager or Cloud-managed Symantec Endpoint Security (ICDm).  Depending on the management platform, it may enable additional features or protection.  The naming conventions between the two platforms for similar agent functionality can differ slightly.  For example, the On-Premise “Application and Device Control policy” is the same as the Cloud-managed Custom Application Behavior Policy” and “Device Control Policy”.  As a result, some of the steps below may be slightly different depending on how the agent is managed.  The general methodology remains the same however; , remove variables from the equation until the cause is identified.

Uninstall the SEP or SES agent

This step can often be overlooked depending on knowledge of the issue.  This completely removes the agent from the system and requires a restart.  It is useful in situations where the issue is intermittent or if it is unclear whether or not the SEP/SES agent is a contributing factor.

Steps to manually uninstall the agent

  1. Click on the Start menu, then open the Control Panel.
    • In the search text box type Control Panel, then click it.
  2. Under Programs, click Uninstall a program.
  3. Click Symantec Endpoint Protection.
  4. Choose Uninstall.
  5. You may be prompted to enter an uninstall password.
  6. Click Next through the uninstall wizard until it finishes.
  7. Restart the computer.
  8. Verify that the SEP or SES agent has been removed.

After restarting, attempt to reproduce the issue.  

  • If the issue reoccurs, the protection agent is not at fault and should be reinstalled.
  • If the issue is intermittent, wait a reasonable amount of time for the issue to reoccur
    • If the issue cannot be reproduced with the protection agent uninstalled, reinstall the agent and confirm the issue is able to be reproduced again, then continue testing using further steps in this document.

Stopping SMC

Stopping smc (Symantec Management Client) is a quick troubleshooting step that disables several protection features of SEP at once.  If configured, a password prompt will be presented when this command is issued.  For details about which features are disabled when issuing a smc -stop command, visit the KB below.

What functions of the Endpoint Protection client are disabled by the smc -stop command
https://knowledge.broadcom.com/external/article/157972/what-functions-of-the-endpoint-protectio.html

Since this command disables many of the protection capabilities of the agent but leaves the drivers running, it can be a useful first step to guide additional troubleshooting steps.  

  • If stopping smc resolves the issue, often there’s a policy change that may work around the issue as well.  
  • If the issue still occurs with smc stopped, you’ll most likely need to jump to Uninstalling features so the underlying driver(s) is removed as well.

*Note: The SEP agent UI must be closed to stop smc.

To stop smc follow these steps:

  1. Click on Start, type Run
  2. In the Run text box, type “smc -stop” (no quotes), click OK.
  3. Enter the password configured in the SEPM’s Client Password Settings.
  4. Wait a few moments for SEP shield icon to disappear from the client system tray
  5. Test again.

To start smc follow these steps:

  1. Click on Start, type Run.
  2. In the Run text box, type “smc -start” (no quotes), click OK.
  3. Wait a few moments for the SEP shield icon to reappear.

Removing Protection Components or Features

This is the most common and useful step in component isolation troubleshooting.  It involves removing protection features from the device so applicable policies are not enforced and component drivers are removed.

Since the protection agent is made up of several different components such as Malware Protection, Firewall, Intrusion Prevention, Application Control, etc, removing each one by one for testing can be a slow process.  It’s best to take a divide and conquer approach.  Start by removing half of the features, restarting and testing again.  Once you have the results, if the issue still occurs, remove half of the remaining components then test again.  Repeat these steps until the issue has been narrowed down to a specific component.

***Note: There are mechanisms in place to prevent the modification of an agent’s installed feature set.  Depending on the method being used below, ensure you’ve temporarily removed any enforcement of installed features or know the uninstall password.  If this consideration is ignored you may see removed features automatically reinstalled which will prolong or invalidate testing.  These include:

  • Client Password settings configured in SEPM or ICDm
  • Tamper Protection
  • Feature Selection Policy (SES) or Install Package (SEPM)
  • Environmental factors such as GPOs or other third party applications which enforce software installation.

Depending on the issue, you will be able to make an educated guess on which feature(s) may be causing the issue and only remove one or two specific components to test your hypothesis.  Generally speaking, here are some “buckets” of features that often go along with certain types of issues.

Network related issues

Browser related issues

    • Intrusion Prevention
    • Browser Protection feature
    • Advanced Download Protection (under AV/AS, Malware protection)

USB/External devices not working

    • Application and Device Control
    • Device Control

There are a few different ways to remove features from devices. While the specifics vary per method, they all can be effective.  Use whichever method is preferable to you.  The Control Panel method allows the most granular control while the other options may be easier from a management console perspective.

Using Control Panel

This is the easiest and fastest method for rapid testing if you have access to an affected device.

***Note: If you have a Feature Selection Policy (Cloud) or Install Package with maintain existing features (On-prem) assigned to the group which this device is in, any changes directly in the control panel will be reverted shortly after any manual changes are made.  It’s suggested that you move this device to a test group that doesn’t have either of those items applied so it doesn’t interfere with testing.

  1. Click on the Start menu, then open the Control Panel.
  2. In the search text box type Control Panel, then click it.
  3. Under Programs, click Uninstall a program.
  4. Click Symantec Endpoint Protection.
  5. Choose Change, you may be prompted to enter a password.
  6. Click Next in the Installation Wizard, then choose Modify.
  7. On the Custom Setup page, you’ll be presented with features and their current install status.  (Example screenshot below steps)
  8. To remove a feature click the drop down box to the left of the name and choose the “Entire feature will be unavailable” option.
  9. Repeat Step 8 for all features you wish to remove.
  10. Click Next and finish the uninstall wizard.
  11. Restart the computer if prompted.

After restarting, attempt to reproduce the issue.

  • If the issue reoccurs.
    • Remove additional features until the issue can no longer be reproduced.
  • If the issue cannot be reproduced.
    • If multiple features were removed at one time, add individual features back one at a time and test if the issue returns after each feature.
    • If only one feature was removed, you have isolated the issue to this feature and can continue investigating by concentrating on settings within applicable product policy.

Using Feature policy (SES, Cloud only)

If directly accessing the client to remove features using the Control Panel isn’t feasible, the next best option is using a Feature Selection Policy to selectively control which features are installed on an agent.  The Feature Selection policy is an optional policy within SES that, when applied to a Group with devices in it, will control what features those devices have installed.

***Note:  It is not possible to remove some features using this method such as Malware Protection.

You can use this policy to selectively add or remove features from devices without needing remote access.  To use this option:

  1. Login to the SES Security Cloud portal.
  2. Click Devices (left), then Device Groups (tab).
  3. Create a new group for this testing.
  4. Click on Policies (left), click Create Policy button (upper right).
  5. Name the policy, click Add button.
  6. Scroll down to find the Symantec Feature Selection Policy (Feature Selection) template, click Create.
    • There are two sections to the policy: Windows Workstation and Windows Server.  Ensure you’re moving the toggles under the correct section.
  7. Click the Show Features button for either Windows Workstation or Windows Server (depends on the target device)
  8. Uncheck or toggle off the desired features, click Save. (Example screenshot below steps)
  9. Apply the policy to the newly created test group.
  10. Click the Apply Policy button.
  11. Check the box next to your test group, click Next.
  12. On the confirmation page, click Submit if you’re satisfied with the changes.

It will take a few minutes for the devices to retrieve the policy and apply the changes.  Depending on the features removed, a restart may be required.  The device will inform the user if a restart is required, the Security Status in ICDm will also indicate a restart is required.

***Note: Restart settings can be controlled using the System Policy under Client Restart Settings section.  By Default, a logged in user will be able to delay a required restart.

Once restarted, you can confirm the features were removed from the Device by clicking on it and scroll down to the Feature Status section.  After confirming the Feature(s) have been removed, test for the issue again.

Using Install Packages tab (SEP, On-prem only)

If directly accessing the client to remove features using the Control Panel isn’t feasible, the next best option is to use the “Install Packages” feature to selectively control which features are installed on an agent.

  1. Login to the SEPM
  2. Create a test group, name it accordingly.
  3. Click the newly created test group.  On the Policies tab for that group, uncheck “Inherit policies and settings from parent group <group name>”
  4. Click Admin (left), then Install Packages
  5. Click Client Install Feature Set, under Tasks click Add Client Install Feature Set.
  6. Name the Feature Set accordingly.
  7. Uncheck boxes next to the features you want to remove.  Example below removing Advanced Download Protection and Intrusion Prevention. (Example screenshot below steps)
  8. Click OK
  9. Go back to the Clients page, click on the test group, then click the Install Packages tab.
  10. Under Tasks (left), click Add a Client Install Package.
  11. Choose the following items:
    • Select the package…   This should be the same version the test device is now.
    • Version Selection - This should match the device’s current SEP version.
    • Uncheck Maintain existing client features when updating. (this is what triggers the client to use the feature set created in step 5).
    • In the Select the features you want to use, select the Feature Set created in Step 5.
    • Install Settings - Defaults are acceptable.  These settings control things like reboot settings so it’s important to know the configuration of your install settings.
  12. Click OK.
  13. Move test devices to the test group.

Once you have clicked OK the SEPM will make this altered installation package available to any devices in that test group.  Depending on the configuration communication settings, the client will check in with the SEPM and then see that it has been given installation instructions that differ from what it currently has installed.  It will download any necessary installation files from the SEPM (most likely they’re already cached) and then perform a reconfiguration of the installed features.  After it has removed or added features based on the Feature Set policy, the client may need a restart.  

After the client has removed or added the desired features and a restart has been performed (if needed), attempt to reproduce the issue again.  Repeat these steps as necessary until the Feature causing the issue is found.

Disabling feature by policy or withdrawing policy

If you’re fairly certain which feature is causing the issue, but all attempts to make configuration changes to resolve the issue have failed, a good troubleshooting step is to either disable the feature by policy or withdraw the policy.  This is an excellent confirmation step because if taking this action resolves the issue it confirms you’ve successfully isolated the issue to one feature.  In most cases where disabling the policy feature resolves the issue, a combination of policy changes and/or exceptions can resolve the issue.

Disabling a feature by policy or withdrawing the policy means that the underlying driver or technology is still in place, it just won’t be enforcing any policies.

The specific steps below will vary depending on the policy in question.  But we’ll cover the most common situations.  Some policies can be withdrawn from a group entirely, others cannot.  Some policies have a simple on/off toggle for the entire feature, others do not.  So the steps may vary slightly depending on the policy/feature you’re working with but we’ll cover the most common steps.

Steps for Symantec Endpoint Protection Manager (On-prem)

***Note: Create a group specifically for testing so only target test devices are impacted.

Steps for Withdrawing a policy (on-prem)

Not all policies can be withdrawn from a Group, but for those that can, the steps are as follows. 

Policies that cannot be withdrawn:

    • Virus and Spyware Protection
    • Memory Exploit Mitigation
    • LiveUpdate
  1. Login to the SEPM, click on Clients (left), then select the test group with test devices in it.
  2. Click on the Policies (tab), uncheck “Inherit policies and settings…” if it isn’t already.
  3. This will allow the modification of applied policies specifically to the test group
  4. To the right of the policy you want to remove, click Tasks then Withdraw Policy.
  5. Click Yes.

This will remove the policy from the group and any devices contained within it.  Wait for the client to receive the updated policy change and then test again.

Steps to disable a policy if it cannot be Withdrawn

****Note: Create a copy of any policy you intend to test disabling and apply it to the test group.  This way only the desired test devices are impacted.  Also disable inheritance to allow modification of policies for this test group.

Uncheck “Enable this policy”

This is the easiest method to completely disable a policy, however not all policies can be disabled this way.  To determine or not the policy you need to disable can be modified this way, follow these steps:

***Note: Only disable test policies using this method so the entire environment doesn’t have the feature disabled. 

  1. Login to the SEPM, click on Policies (left).
  2. Choose the policy type, then click the desired test policy.
  3. Under Tasks, choose Edit the policy.
  4. On the policy Overview (left) page, uncheck Enable this policy.
  5. Click OK.

This will disable the policy for any group it’s assigned to and any devices contained within it.  Wait for the client to receive the updated policy change and then test again.

Disabling or Unchecking specific settings within a policy.

For policies (Virus and Spyware, LiveUpdate, and Memory Exploit Mitigation)  that have the “Enable this policy” checkbox greyed out, this is the next best method for disabling features by policy.

For Memory Exploit Mitigation

To disable Memory Exploit Mitigation (MEM) by policy:

  1. Edit the MEM policy
  2. Click on Memory Exploit Mitigation (left)
  3. Uncheck “Enable Memory Exploit Mitigation”
  4. Click OK to save the changes.

Test to see if the issue can be reproduced.

Another option rather than disabling may be to check the “Set the protection action for all techniques to log only”.  This would essentially place the system in a monitor mode.

For Virus and Spyware Protection Policy

***Note: If the issue is with AV/AS, most likely it’s Auto-Protect, SONAR, Download Protection or ELAM.  Rarely is it one of the other features.  All of the settings are mentioned below, but you can save time by focusing on the four items mentioned above.

Edit the policy then make the following changes:

  1. Under Scheduled Scans
    • Administrator-Defined Scans: Uncheck any “Enabled” checkbox.
    • On the Advanced tab, uncheck “Run an Active Scan when new definitions arrive”
  2. Under each Protection Technology, click each item and make the following changes.
    • Auto-Protect: Uncheck “Enable Auto-Protect”
    • Download Protection:  Uncheck “Enable Download Insight…”
    • SONAR
      • High risk detection: Set to Log
      • Low risk detection: Set too log
      • DNS change detected and Host file change detected: Set both to Log
      • Network Settings: Uncheck “Scan files on remote computers”
    • Early Launch Anti-Malware Driver: Uncheck “Enable Symantec early launch anti-malware”
  3. Under each Email Scans section
    • Uncheck each respective “Enable <feature> Auto-Protect”
  4. Save the policy.

Test to see if the issue can be reproduced.

For LiveUpdate Policy

It’s unlikely the policy itself is causing an issue.  It merely controls when and where the agent gets content (definitions).  If you need to stop a client from running LiveUpdate however, follow these steps:

  1. Edit the LiveUpdate policy, choose Server Settings under Windows
    • Uncheck every checkbox on this page.

This can only be done for Windows agents.  This will prevent Windows devices from running LiveUpdate or getting content from the SEPM.

Test to see if the issue can be reproduced.

Steps for Cloud

It’s recommended to create a “Component Isolation” test group directly under the “Default” group. It will simplify testing.

While most policies can be withdrawn from groups, if there’s a different policy of the same type applied to the parent group, that policy will then be inherited, invalidating the testing.  To get around this, the removal testing would be done in a group under the Default group, and the policies are also removed from that group.  In this scenario it would impact other child groups which is undesirable.  Therefore, the least destructive method is creating a new policy of the same policy type you’re trying to remove and then “disable the feature” by policy.  Disabling the feature by policy will vary per policy type.  Below are the steps per policy:

For all policy types, you must create a new policy first:

  1. Login to SES Cloud
  2. Click on the Policies tab, Click Create Policy button (upper right)
  3. Name the policy some descriptive like “Isolation testing - <policy type>”, click Add button
  4. Select the Policy Type you need to test with, click Create.
  5. Edit the policy per the instructions below for the Policy Type.
  6. Save the policy.
  7. Apply the policy to the test group the test device(s) are in.

 

Adaptive Protection

    1. All actions will be set to Monitor by default.
    2. Turn off Auto Tune.

Allow List

    1. It is unlikely this policy has a negative impact on the agent.  It’s more likely that removing any existing exceptions would be detrimental to testing.

App Control

    1. Under General Settings, enable “Allow Symantec Trusted”.
    2. Under Enforcement Settings, select “Monitor Only”.

Compliance

    1. Under Scheduling, select “Never run the Host Integrity check”.

Custom Application Behavior

    1. Under Custom Rule Sets, ensure Status is Off for all Rules.

Deny List

    1. Ensure the policy has no items added.

Detection and Response

    1. Under Endpoint Activity Recorder Status, turn it off (grey toggle status).

Device Control

    1. Ensure there are no Devices under Device Control Rules.

Exploit Protection

    1. Under General Settings, toggle on “Run in monitor mode”.

Feature Selection

    1. This policy shouldn’t be modified unless you’re isolating components by uninstalling them.

Firewall

    1. Under General Settings, toggle “Firewall” to the off position (grey).

If the issue is narrowed down to the Firewall component, testing Firewall Rules is the next step.  For additional steps see article Network application does not work with Endpoint Protection firewall installed.


Intrusion Prevention

    1. Scroll down to Advanced Settings, toggle off “Enable URL reputation”
    2. Click the Show Advanced button, turn on “Monitor Mode” for Intrusion Prevention and Browser Protection.

Network Integrity

    1. On the VPN Based tab, toggle off “Network protection”

Threat Defense for AD

*Note: If TDAD was never configured, this policy can be ignored.

    1. Under General Settings, toggle off “Enable Policy”

Trusted Updater

    1. This policy can be ignored for component isolation testing

Unwanted Mobile Application

    1. Only applies to iOS/Android, can be ignored for this windows testing.

Web and Cloud Access Protection

    1. Toggle “Default Web and Cloud Access Protection Policy” to Off.


For Reference:

  • Policies that can be Removed
    • Adaptive Protection
    • Allow List
    • App Control
    • Compliance
    • Custom Application Behavior
    • Deny List
    • Detection and Response
    • Exploit Protection
    • Feature Selection
    • Memory Exploit Mitigation
    • Network Integrity
    • Threat Defense for AD
    • Trusted Updater
    • Unwanted Mobile Application
    • Web and Cloud Access Protection
  • Policies that cannot be Removed
    • System
    • Malware Protection
    • Intrusion Prevention
    • Device Control
    • Firewall

Disabling remaining drivers in Core Files only

If all SEP/SES protection features have been uninstalled using the steps above, the install is in a “Core Files Only” mode.  This refers to the Interactive SEP Custom Setup page where you can remove all features except for Core Files.  After reducing the install set to Core Files only and restarting, if you’re still experiencing the issue, there are several drivers remaining when in Core Files only.  They are listed below along with steps to disable them if necessary.

These drivers can be disabled using Windows Service Configuration command via command prompt or by editing the registry.  Use whichever is most convenient to you, but document the current state so it can be reverted when testing is complete.

*Note: Both of these methods require Tamper Protection be disabled.  Only disable Tamper Protection on test systems.

Below is a list of Core Files only drivers to test disabling using one of the methods below the list.  The list is ordered by likelihood of causing an issue.

  • BHDrvx64
  • SymNets
  • SymEFASI
    • To disable SymEFASI you must rename C:\Windows\System32\drivers\symefasi\0705030.037\symefasi64.sys to .bak and then restart the system.
  • SymELAM (most likely if the issue is at startup)
  • SymEvent
  • SymIRON
  • EraserUtilRebootDrv
  • eeCtrl

Use the service name above in place of <service_name> below in the steps.  No quotes, no greater or less than symbols.

*Note: Tamper Protection must be disabled to use either method listed below.  Only disable Tamper Protection on devices these steps will be performed on.

To disable a service in the registry

  1. Open regedit
  2. Navigate to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
  3. Locate the desired service, click on it
  4. Find the Start DWORD for the service, document the current value
  5. Change the value to 4, restart the system.
  6. Test again.

To disable a service using Service Configuration (sc)

  1. Open an Administrator elevated Command Prompt
  2. Type “sc qc <service_name>” hit enter
  3. Note the START_TYPE
  4. To disable the service, type “sc config <service_name> start= disabled”, hit enter
  5. Restart the system and test again.

Depending on the driver identified as causing the issue, it may be possible to leave it disabled while reinstalling protection features so that some level of protection is applied to the device while the investigation is ongoing.

Once the issue has been narrowed down to a specific service, revert the previously changed service settings.

Additional Information

Powered by Wolken Software