Windows Server 2008 with IIS 7: HTTP Status Code 503.2 in IIS log; Symantec Endpoint Protection (SEP) clients randomly disconnecting and reconnecting.

Products

Endpoint Protection

Issue/Introduction

Machines with a managed SEP client installed randomly disconnect from, and reconnect to, the Symantec Endpoint Protection Manager (SEPM) for no apparent reason.

Symptoms

The SEPM has been installed to a Windows Server 2008 machine. Communication between the manager and the clients has been established. After a short period of time, it becomes noticable that clients are disconnecting and reconnecting to the manager (the green dot on the shield icon in the system tray disappears and reappears). In the SCM-SERVER-0.LOG file, an error similar to the following one appears:

2009-10-09 09:32:05.078 SEVERE: Unknown Exception in: com.sygate.scm.server.task.ClientTransportInfoTask
java.lang.Exception: HTTP 503 Service Unavailable, URL: http://localhost/secars/secars.dll?action=36
at com.sygate.scm.common.communicate.Communicator.sendRequest(Communicator.java:303)
at com.sygate.scm.server.task.ClientTransportInfoTask.run(ClientTransportInfoTask.java:116)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
com.sygate.scm.common.communicate.CommunicationException: Unexpected server error. ErrorCode: 0x10010000
at com.sygate.scm.common.communicate.Communicator.sendRequest(Communicator.java:324)
at com.sygate.scm.server.task.ClientTransportInfoTask.run(ClientTransportInfoTask.java:116)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2009-10-09 09:32:08.683 SEVERE: IISCacheTask connect to secars failed: Service Unavailable

In the IIS logs, you may find an HTTP code similar to the following one:

Smc 503 2 0 1

In the SEPM, errors with the following verbage may appear:

"Unexpected exception...0x10010000..."

Variables that may contribute to the situation:

SEPM is installed to a Windows Server 2008 machine, which is running IIS 7
There are a large number of machines in the environment; 5000+ machines
SEPM is configured for the PUSH communication mode

Cause

The HTTP code 503.2 is specific to IIS 7. The cause for this code is "concurrent request limit exceeded." According to Microsoft, "The appConcurrentRequestLimit property is set to a value that is lower than the current number of concurrent requests. IIS 7.0 does not allow more concurrent requests than the value of the appConcurrentRequestLimit property."

Resolution

To properly troubleshoot this problem, you will need to have logging enabled in IIS 7 for the website that the SEPM components are installed to. For SEPM installations with MR3 and newer, this website is Symantec Web Server.

Follow these instructions to enable logging in IIS

How to turn on Logging in IIS 7:

Open up the Server Manager
1. Click Start
2. Type Server Manager in the search bar
3. Click Server Manager under Programs at the top-left of the start menu
In the left-pane expand Roles
Then expand Web Server (IIS)
Highlight Internet Information Services (IIS) Manager
In the next pane to your right under the Connections header expand the server name
Expand Sites
Expand Symantec Web Server
Select the first virtual directory under the web site.
In the right-pane under the IIS heading double-click Logging
Be sure that logging is set to daily and take note of the directory it is set to.
On the right-hand side, under Actions, click Enable to start logging.
Enable logging for the remaining virtual directories

If everything is grayed out here you will need to install the logging option of IIS.

To install the logging feature for IIS do the following:
1. In the far left-pane, under Roles, click on Web Server (IIS)
2. In the right-pane, next to the heading Role Services, click Add Role Services
3. Under Health and Diagnostics put a check next to HTTP Logging
4. Click the Install button
  
  NOTE: A reboot is required before changes will take effect

Logs are located by default in the \inetpub\logs\LogFiles directory

With IIS logging enabled, open the LOG file with today's "modified" date. Look for a code similar to this one: Smc 503 2 0 1. If you are receiving a 503.2 code, then the server is hitting the "concurrent request limit" threshold.

- - - - - -

To resolve this problem, you will need to increase the value of the appConcurrentRequestLimit property. The default value is 5000.

This property is controlled by the <serverRuntime> element. There is no User Interface (UI) for this element. Any changes must be performed using Command Prompt. Speak with your system administrator, contact Microsoft, or refer to the documents in the References section of this article for information and instructions on how to change the value of the appConcurrentRequestLimit property.

- - - - - -

An alternative to changing the appConcurrentRequestLimit value would be to change the client communication mode from push to pull.

Difference Between "Push" and "Pull" Communication Modes:

Clients that use Push Mode download policies and content as soon as they become available. With Push Mode, a connection is kept open so that the manager can contact the client immediately when data is available. Using this mode requires greater network bandwidth, but ensures that clients receive policy and content updates immediately.

Clients that use Pull Mode download policies and content based on the Heartbeat Interval setting, which is set to 5 minutes by default. In Pull Mode, a connection is established at every heartbeat and terminated as soon as the required communication is over. This mode uses less network bandwidth, but clients do not receive policy and content updates until they connect to the manager to retrieve the data.

Steps to change the communication mode in client groups

References

Support.Microsoft.com: The HTTP Status Codes in IIS 7.0

MSDN.Microsoft.com: IIS 7.0: serverRuntime Element (IIS Settings Schema)

IIS.net: Server Runtime <serverRuntime>