Endpoint Protection Manager stops delivering definition deltas after java.lang.OutOfMemoryError errors
search cancel

Endpoint Protection Manager stops delivering definition deltas after java.lang.OutOfMemoryError errors

book

Article ID: 159349

calendar_today

Updated On:

Products

Endpoint Protection

Issue/Introduction

You notice that a large number of clients begin requesting full.zip definition downloads from the Symantec Endpoint Protection Manager (SEPM), which results in network performance issues. There appears to be no definition corruption on the clients.  You may also notice that clients are failing to register with the SEPM.  

Shortly before the issue starts, you notice the following error in scm-server-0.log

2014-05-28 12:04:52.127 THREAD 34 SEVERE:  in: com.sygate.scm.server.task.ClientTransportInfoTask
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)

You may also notice the following error in catalina.err:

Exception in thread "http-bio-9090-Acceptor-0" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:693)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:176)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:156)
at org.apache.catalina.core.StandardThreadExecutor.execute(StandardThreadExecutor.java:168)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:525)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:230)
at java.lang.Thread.run(Thread.java:724)

Cause

The OutOfMemory errors listed above indicate that Tomcat does not have enough native memory available to generate a new thread. The Tomcat engine will open a new thread to process incoming HTTP requests. This results in the Tomcat HTTP acceptor(http-bio-9090-Acceptor) no longer processing incoming requests on 9090. This can include Delta requests for definitions, insertion of definitions into the database (either through LiveUpdate or manual JDB insertion), and client registrations.  

Resolution

Symantec is currently investigating this issue. As a workaround, you can perform the following performance tuning.

  1. Reduce the Java heap size for Semsvc process.  This will increase the amount of available native memory. 
    1. Stop the Symantec Endpoint Protection Manager services.
    2. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\semsrv\Parameters.
    3. Modify the following values:
      • JVM Option 0 as "-Xms256m".
      • JVM Option 5 from "-XX:MaxPermSize=256m" as "-XX:MaxPermSize=128m"
    4. Add a new String value, "JVM Option Number 17", the value is "-Xss128k".
    5. Modify "JVM Option Count", select Decimal, and increase the value as 18(17+1).
    6. Restart the Symantec Endpoint Protection Manager services.

      *Note*  The default value for JVM Option 0 will vary depending on the initial install settings.  For default values, see "Determining the default settings for the network sizes that you select during installation of the Symantec Endpoint Protection Manager."  
       
  2. Reduce the AgentInfoTask thread count.  This will decrease the number of threads used to process AgentInfo logs.    
    1. Stop the Symantec Endpoint Protection Manager services.  
    2. Add the following line into conf.properties: scm.agentinfo.threadcount=4
    3. Restart the Symantec Endpoint Protection Manager services.
       
  3. Reduce the size of the SEPM schema XML cache.  
    1. Stop the Symantec Endpoint Protection Manager services.
    2. Add the following line to conf.properties: scm.cache.thereshold=600  (Yes, this is spelled correctly.) 
    3. Restart the Symantec Endpoint Protection Manager services.
       
  4. Add the SQL timeout settings in the ROOT.xml (This step was implemented in 12.1.5 RU5)
    1. Edit SEPM_INSTALL\tomcat\conf\Catalina\localhost\ROOT.xml.
    2. Update attribute "url" add "socketTimeout=600" as below:
      • Modify "jdbc:jtds:sqlserver://sql-server-machine:1433/sem5" as "jdbc:jtds:sqlserver://sql-server-machine:1433/sem5;socketTimeout=600"
    3. Restart the Symantec Endpoint Protection Manager services.
Powered by Wolken Software