High usage SEPMs report multiple "unexpected server error" events in the scm-server-0.log
Article ID: 156637
Updated On:
Products
Issue/Introduction
The Symantec Endpoint Protection Manager (SEPM) reports multiple unexpected server errors in the scm-server log. The issue is more apparent on large environments with several thousand clients that utilize teamed NICs.
2012-06-05 09:02:57.817 THREAD 30 SEVERE: in: com.sygate.scm.server.task.ClientTransportInfoTask
Cause
The ClientTransportInfoTask is a query of secars to test throughput and server health. The issue happens when a connection cannot be established to Apache via the loopback address. Essentially, the server attempts to connect to itself for a status check and finds that there are no available connections. There are multiple causes for this logging. The issue appears as though Windows has run out of available connections. However, per Apache documentation, this issue can occur if the IP address needs to be renegotiated after the services have already started up. That, or Apache has indeed run out of available connections.
Resolution
Solution #1:
If Apache is run on a server with teamed NICs, this issue may occur when Apache attempts to renegotiate the IP address after the service has already started. If you experience this issue on a SEPM with teamed NICs, you can bind Apache to a specific IP address in order to resolve the issue.
1. Stop the Symantec Endpoint Protection Manager and Symantec Endpoint Protection Webserver services.
2. Edit the httpd.conf file in \Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\apache\conf\httpd.conf
Bind Apache to the specific IP addresses on the system by changing the following line:
Listen 8014
Replace it with:
Listen sepmIPaddress:8014 (Where 'sepmIPaddress' is the IP address of the SEPM)
Listen 127.0.0.1: 8014
3. Restart the Symantec Endpoint Protection Manager and Symantec Endpoint Protection Webserver services.
Applies To
Server 2008 Standard x64 SEPM
12.1.1000 connected to SQL Server 2008R2.
20,000+ clients per SEPM
Broadcom Teamed NICS
Broadcom NetXtreme II Driver: 14.4.8.4
BASP (Broadcom Advanced Server Program Driver) version: 1.4.6
Solution #2:
The "ClientTransportInfoTask" error will also be present when Apache has run out of available threads for clients checking in to the SEPM. This can happen when the SEPM has too many clients checking in. The SEPM by default will accept 3000 concurrent connections from clients. When this maximum is reached, the following similar logging will be present within the \SEPM\Apache\logs\<timezone-error.log>:
Thread(xxxx) Contexts(Total: 3007 Sync: 3000 Async: 7 Max: 3000) returning null context
When the Total exceeds the Max (3000), Clients will fail to check in to the SEPM. To resolve this issue, Apache may be configured to accept more than 3000 connections. Increasing the number of connections will increase load on the SEPM, so make sure your hardware can handle potentially twice as many clients checking in. If increasing the connections puts too much stress on the SEPM, consider adding another SEPM for loadbalancing purposes.
How-to edit the apache connections limit:
This is done by modifying the httpd.conf file (SEPM\apache\conf\httpd.conf), line 700:
# Max number of connection requests that will be queued in the child httpd process
# ConnectionsToQueuePerChild <number of connections>
# +--> Must be greater than ThreadsPerChild item.
# Default is 5000.
ConnectionsToQueuePerChild 3000
Edit the "ConnectionsToQueuePerChild" value from 3000 up to 6000 and save the httpd.conf file. The SEPM service must be restarted for the change to take effect.
Solution #3
This error can also be caused by having the incorrect webserver port specified in conf.properties. Open conf.properties in the folder C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\tomcat\etc. Navigate to the setting
scm.webserver.http.port
This should read "scm.webserver.http.port=8014" (no quotes) by default. If you've customized this port using the Management Server Configuration Wizard then it may be slightly different. Changing this to the incorrect port will result in ClientTransportInfoTask errors in the scm.server-#.log(s).
Feedback
