High usage SEPMs report multiple "unexpected server error" events in the scm-server-0.log

book

Article ID: 156637

calendar_today

Updated On:

Products

Endpoint Protection

Issue/Introduction

The Symantec Endpoint Protection Manager (SEPM) reports multiple unexpected server errors in the scm-server log.  The issue is more apparent on large environments with several thousand clients that utilize teamed NICs.  

 2012-06-05 09:02:57.817 THREAD 30 SEVERE:  in: com.sygate.scm.server.task.ClientTransportInfoTask

java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at com.sygate.scm.common.communicate.Communicator.getResponse(Communicator.java:1345)
at com.sygate.scm.common.communicate.Communicator.getResponse(Communicator.java:878)
at com.sygate.scm.server.task.ClientTransportInfoTask.execute(ClientTransportInfoTask.java:83)
at com.sygate.scm.server.task.MonitoredTimerTask.run(MonitoredTimerTask.java:22)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
com.sygate.scm.common.communicate.CommunicationException: Failed to connect to the server.
 
Make sure that the server is running and your session has not timed out.
If you can reach the server but cannot log on, make sure that you provided the correct parameters.
If you are experiencing network issues, contact your system administrator. ErrorCode: 0x80020000
at com.sygate.scm.common.communicate.Communicator.getResponse(Communicator.java:1382)
at com.sygate.scm.common.communicate.Communicator.getResponse(Communicator.java:878)
at com.sygate.scm.server.task.ClientTransportInfoTask.execute(ClientTransportInfoTask.java:83)
at com.sygate.scm.server.task.MonitoredTimerTask.run(MonitoredTimerTask.java:22)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Cause

The ClientTransportInfoTask is a query of secars to test throughput and server health.  The issue happens when a connection cannot be established to Apache via the loopback address.  Essentially, the server attempts to connect to itself for a status check and finds that there are no available connections.  There are multiple causes for this logging. The issue appears as though Windows has run out of available connections.  However, per Apache documentation, this issue can occur if the IP address needs to be renegotiated after the services have already started up. That, or Apache has indeed run out of available connections.

Resolution

Solution #1:
If Apache is run on a server with teamed NICs, this issue may occur when Apache attempts to renegotiate the IP address after the service has already started.  If you experience this issue on a SEPM with teamed NICs, you can bind Apache to a specific IP address in order to resolve the issue.  

1.  Stop the Symantec Endpoint Protection Manager and Symantec Endpoint Protection Webserver services.  

2.  Edit the httpd.conf file in \Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\apache\conf\httpd.conf

Bind Apache to  the specific IP addresses on the system by changing the following line:

Listen 8014

Replace it with:

Listen sepmIPaddress:8014 (Where 'sepmIPaddress' is the IP address of the SEPM)

Listen 127.0.0.1: 8014

3.  Restart the Symantec Endpoint Protection Manager and Symantec Endpoint Protection Webserver services.  

 

Applies To

Server 2008 Standard x64 SEPM

12.1.1000 connected to SQL Server 2008R2.  

20,000+ clients per SEPM

 

Broadcom Teamed NICS

Broadcom NetXtreme II Driver: 14.4.8.4

BASP (Broadcom Advanced Server Program Driver) version: 1.4.6

Solution #2:
The "ClientTransportInfoTask" error will also be present when Apache has run out of available threads for clients checking in to the SEPM.  This can happen when the SEPM has too many clients checking in.  The SEPM by default will accept 3000 concurrent connections from clients.  When this maximum is reached, the following similar logging will be present within the \SEPM\Apache\logs\<timezone-error.log>:

Thread(xxxx) Contexts(Total: 3007 Sync: 3000 Async: 7 Max: 3000) returning null context

When the Total exceeds the Max (3000), Clients will fail to check in to the SEPM.  To resolve this issue, Apache may be configured to accept more than 3000 connections.  Increasing the number of connections will increase load on the SEPM, so make sure your hardware can handle potentially twice as many clients checking in.  If increasing the connections puts too much stress on the SEPM, consider adding another SEPM for loadbalancing purposes.

How-to edit the apache connections limit:
This is done by modifying the httpd.conf file (SEPM\apache\conf\httpd.conf),  line 700:
# Max number of connection requests that will be queued in the child httpd process
#    ConnectionsToQueuePerChild <number of connections>
#                                       +--> Must be greater than ThreadsPerChild item.
#                                            Default is 5000.
ConnectionsToQueuePerChild 3000

Edit the "ConnectionsToQueuePerChild" value from 3000 up to 6000 and save the httpd.conf file. The SEPM service must be restarted for the change to take effect.

 

Solution #3

This error can also be caused by having the incorrect webserver port specified in conf.properties.  Open conf.properties in the folder C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\tomcat\etc.  Navigate to the setting

scm.webserver.http.port

This should read "scm.webserver.http.port=8014" (no quotes) by default.  If you've customized this port using the Management Server Configuration Wizard then it may be slightly different.  Changing this to the incorrect port will result in ClientTransportInfoTask errors in the scm.server-#.log(s).