Policy Server doesn't connect back to the first ODBC Session Store

book

Article ID: 130316

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On SITEMINDER CA Single Sign On Federation (SiteMinder)

Issue/Introduction

 

When running a Policy Server with several ODBC Session Stores, the
ODBC connection might be seen as unequally distributed among the ODBC
instances. One might want to know how it's possible to get connection
equally distributed among the Session Store instances.

 

Environment

 

  Policy Server all versions;
  2 Session Stores on ODBC Oracle; 

 

Resolution

 

Even if the ODBC driver allows the usage of "alternateservers" and
"loadbalancing" features, these will proceed only with failover, even
if the feature is called "loadbalancing". The value of "loadbalancing"
determine the order in which the failover occur on the configuration
(in serie or random)(1)(2).

In order to avoid trailing connections to down ODBC instances, in
system_odbc.ini, configure the following :

  - Set the KeepAlive parameter to 1 in order to 
    close any connection to a down server (3)(4)(5). 

This will avoid the connections to be kept open to ODBC instances
which are down.

More, you can also configure 2 separated data sources and remove the
configuration of alternateservers and loadbalancing from the driver
configuration will make all the odbc connection going back to the
Primary Session Store. Note that from documentation, we support
failover only, and the driver loadbalancing configuration stands for
failover too.

Here we've removed the alternateservers and LoadBalancing
configuration and we've defined 2 separate Session Store data sources,
1 for the Primary and another one for the Secondary.

system_odbc.ini : 

  [Session Primary] 
  Driver=/opt/CA/siteminder/odbc/lib/NSora27.so 
  Description=DataDirect 7.1 Oracle Wire Protocol 
  HostName=duspa01-u161157.ca.com 
  PortNumber=1521 
  LoginID=SMUSER 
  Password=xxxxx
  ServiceName=XE 
  #SID=XE 
  CatalogOptions=0 
  ProcedureRetResults=0 
  EnableDescribeParam=0 
  EnableStaticCursorsForLongData=0 
  ApplicationUsingThreads=1 
  DMCleanup=2 

  [Session Secondary] 
  Driver=/opt/CA/siteminder/odbc/lib/NSora27.so 
  Description=DataDirect 7.1 Oracle Wire Protocol 
  HostName=duspa01-u161156.ca.com 
  PortNumber=1521 
  LoginID=SMUSER 
  Password=xxxxx
  ServiceName=XE 
  #SID=XE 
  CatalogOptions=0 
  ProcedureRetResults=0 
  EnableDescribeParam=0 
  EnableStaticCursorsForLongData=0 
  ApplicationUsingThreads=1 
  DMCleanup=2 

In the SmConsole, we've configured the Session Store connection
putting each of the datasources, separated with a coma "," :

sm.registry : 

  HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\Database\SessionServer=235352101 
  Data Source=Session Primary, Session Secondary;    REG_SZ 
  Enabled= 0x1;    REG_DWORD 
  MaxConnections= 0x20;    REG_DWORD 
  Password= {RC2}cJhuVJ9LzQf12y9hk6FdGc9ztDo+Y8Em;    REG_SZ 
  ProviderNamespace= ODBC:;    REG_SZ 
  Use Default= 0;    REG_DWORD 
  User Name= SMUSER;    REG_SZ 

Once done, we've started Policy Server, and we've seen it using the 
Primary Session Store. When the Primary Session Store went off line, 
Policy Server moved connections to the Secondary Session Store. When 
the Primary Session Store came back online, then Policy Server moved 
back the connections to the Primary Session Store : 

smps.log : 

  [13473/4151690960][Fri Dec 28 2018 
  03:08:08][CServer.cpp:4951][INFO][sm-Server-01430] Logging and Trace 
  output in local time. TimeZone: [GMT-5:00]. Daylight Savings: 0 

  [13473/3799419760][Fri Dec 28 2018 
  03:08:12][CSmDbSessionManager.cpp:585][INFO][sm-Server-04350] Using 
  ODBC 'Session Data' data source 'Session Primary'. 

  [13473/4151690960][Fri Dec 28 2018 
  03:08:12][SmSSInDBStore.cpp:299][INFO][sm_LoginLogout_02014] 
  Initialized Session Server. Schema version is 3, lVersion 

  [13473/3715414896][Fri Dec 28 2018 
  03:09:12][CSmDbSessionManager.cpp:180][INFO][sm-Server-04350] Using 
  ODBC 'Session Data' data source 'Session Primary'. 

  [13473/4065667952][Fri Dec 28 2018 
  03:17:38][CSmDbSessionManager.cpp:147][INFO][sm-Server-04330] 
  Failing over to 'Session Data' data source 'Session 
  Secondary'. 

  [13473/3715414896][Fri Dec 28 2018 
  03:22:12][CSmDbSessionManager.cpp:155][INFO][sm-Server-04340] 
  Failing back to 'Session Data' data source 'Session 
  Primary'. 

Netstat output during the use case reproduction : 

We start the Policy Server : 

  Fri Dec 28 03:08:13 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED 
  12171/xe_pmon_XE 

  tcp 0 0 10.162.31.240:31701 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

We put load and Policy Servers build connections to the Primary 
Session Store (.240): 

  Fri Dec 28 03:16:07 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:59662 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:32757 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED 
  12171/xe_pmon_XE 

  tcp 0 0 10.162.31.240:20816 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:22829 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:31004 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 689 10.162.31.240:31701 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:65377 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

We shut down the Primary Session Server (.240) : 

  Fri Dec 28 03:17:37 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 1 0 10.162.31.240:47888 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

  tcp 1 0 10.162.31.240:59662 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED 
  12171/xe_pmon_XE 

  tcp 1 0 10.162.31.240:20816 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

  tcp 1 0 10.162.31.240:22829 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

  tcp 1 0 10.162.31.240:31004 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

  tcp 1 0 10.162.31.240:65377 10.162.31.240:1521 CLOSE_WAIT 
  13473/smpolicysrv 

Policy Server fails over connections to the Secondary Session Store 
(.106) : 

  Fri Dec 28 03:18:38 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

We start over the Primary Session Server : 

  Fri Dec 28 03:21:32 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED 
  18172/xe_pmon_XE 

  tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

Policy Server notices it and starts to move back all connections to
the Primary Session Store :

  Fri Dec 28 03:21:41 EST 2018 

  tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED 
  18172/xe_pmon_XE 

  tcp 0 0 10.162.31.240:39907 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:55979 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:27555 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED 
  13473/smpolicysrv 

Until all connections are back to the Primary Session Store : 

  Fri Dec 28 03:21:48 EST 2018 

  tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED 
  18172/xe_pmon_XE 

  tcp 0 0 10.162.31.240:39907 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:14352 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:36902 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 679 10.162.31.240:55979 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:27776 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:51390 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:27555 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

  tcp 0 0 10.162.31.240:31303 10.162.31.240:1521 ESTABLISHED 
  13473/smpolicysrv 

As stated earlier in this case, you won't be able to achieve
loadbalancing between the Session Stores. So you will never get
connections equally distributed between nodes.

As our documentation states, Session Stores instances should be set as
failover (6).

 

Additional Information

 

(1)

    Configure an Oracle Data Source for CA Single Sign-On 

      AlternateServers= 

      If the primary server is not accepting connections, specifies the
      connection failover to the other Oracle nodes.

      Example:
      (HostName=nete_servername2:PortNumber=1521:ServiceName=nete_servicename[,...])

      LoadBalancing=1 

      Turns on client load balancing, which helps to distribute new
      connections to keep RAC nodes from being overwhelmed with
      connection requests. When enabled, the order in which primary and
      alternate database servers are accessed is random.

    https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/installing/install-a-policy-server/configure-odbc-databases-as-policy-session-key-and-audit-stores/configure-odbc-databases-as-session-store/store-session-information-in-oracle.html

(2)

    Policy Server and ODBC Store inconsistent loadbalancing connections

      Even if the nomenclature of the configuration suggest there's a
      loadbalancing, the following configuration means failover, and
      not loadbalancing (1).

       AlternateServers=(HostName=myserver:PortNumber=1111:ServiceName=myserver.mydomain.com) 
       LoadBalancing=1 
    
    https://knowledge.broadcom.com/external/article?articleId=119929

(3)

    Policy Server Oracle Server Wire Protocol configuration

       KeepAlive=0 

    https://knowledge.broadcom.com/external/article?articleId=32766

(4)

    TCP Keep Alive 

      KeepAlive (KA) 
      If set to 0 (Disabled), the driver does not enable TCPKeepAlive. 
      If set to 1 (Enabled), the driver enables TCPKeepAlive. 

    https://media.datadirect.com/download/docs/odbc/allodbc/index.html#page/odbc/tcp-keep-alive.html 

(5)

    Load Balance Timeout 

      Specifies the number of seconds to keep inactive connections
      open in a connection pool. An inactive connection is a database
      session that is not associated with an ODBC connection handle,
      that is, a connection in the pool that is not in use by an
      application.

    https://media.datadirect.com/download/docs/odbc/allodbc/index.html#page/odbc%2Fload-balance-timeout.html%23

(6)

    Policy Server to Session Store Communication 

      Allows for failover. If a primary session store fails, Policy Servers 
      failover to a secondary session store.

    https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/implementing/implementing-siteminder/architectural-use-cases.html