SiteMinder 500 Errors and Hung/"Ghost" Connections Caused by Firewall-Dropped Idle TCP Sessions Between Web Agent, Access Gateway (SPS), and Policy Server

Products

• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Secure Proxy Server (SiteMinder)
• SITEMINDER
• CA Single Sign-On

Issue/Introduction

When a firewall or network outage silently drops an idle TCP connection between the Policy Server and a Web Agent or Access Gateway (SPS), neither side is notified. The Policy Server keeps the connection in ESTABLISHED state and keeps its worker thread blocked waiting for data that will never arrive [1]. As failed requests accumulate, worker threads stop being released, and clients see intermittent 500 errors, failed authentications/authorizations, growing connection queues, or socket errors such as Failed to send response on session #... in smps.log [1][3].

Cause

TCP has no built-in mechanism to detect a silently-dropped connection unless keep-alive probing is enabled. By default, SiteMinder components rely on OS-level TCP behavior only, so a mid-path firewall reset/timeout goes undetected on both ends [1].

Resolution

Resolution — enable SM_ENABLE_TCP_KEEPALIVE:

Set the environment variable to 1 on the affected component(s) (Policy Server, Web Agent, and/or Access Gateway), then restart the service. Accepted values are 0 (disabled) or 1 (enabled) — anything else disables the feature [1][2].

• - Windows (Policy Server or Web Agent host): System Properties → Advanced → Environment Variables → New → Name: SM_ENABLE_TCP_KEEPALIVE, Value: 1 [4].
• Linux/UNIX (Policy Server or Web Agent host): create and export SM_ENABLE_TCP_KEEPALIVE=1 [2][4].
• Access Gateway (SPS) on Linux: add the export to ca_sps_env.sh in the SPS home directory, then restart [5]:
  
  # cd /{home_sps}/
# source ca_sps_env.sh
# cd /{home_sps}/proxy-engine/
# ./sps-ctl startssl
    
• If a firewall sits specifically between the Agent/Gateway and the Policy Server, set the variable on the agent/gateway side as well, not just the Policy Server [1].

Optional — tune OS keep-alive timing if the default interval is too slow to detect drops in your environment (Linux example):

# cat /proc/sys/net/ipv4/tcp_keepalive_time
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
# cat /proc/sys/net/ipv4/tcp_keepalive_probes

Adjust with echo <value> > /proc/sys/net/ipv4/tcp_keepalive_time, etc. Coordinate with your OS/network team before changing system-wide TCP defaults [3].

Verification:

1. Confirm the environment variable is actually set on the running process (not just the shell/profile).
2. Take a network trace while simulating a firewall drop; keep-alive probes appear as [TCP Keep-Alive] packets. In Wireshark, filter on tcp.analysis.keep_alive to isolate them [3].

Additional Information

1. Error: 500 Web Agent and Policy Server network communication problem
   
   

2. Policy Server Troubleshooting
   
   

3. Verify SM_ENABLE_TCP_KEEPALIVE is working?
   
   

4. SiteMinder: How to set the SM_ENABLE_TCP_KEEPALIVE environment variable
   
   

5. Agent on CA Access Gateway (SPS) loses connection with Policy Server