Policy Server doesn't connect back to the first ODBC Session Store
Article ID: 130316
Updated On:
Products
Issue/Introduction
When running a Policy Server with several ODBC Session Stores, the
ODBC connection might be seen as unequally distributed among the ODBC
instances. One might want to know how it's possible to get connection
equally distributed among the Session Store instances.
Environment
Policy Server all versions;
2 Session Stores on ODBC Oracle;
Resolution
Even if the ODBC driver allows the usage of "alternateservers" and
"loadbalancing" features, these will proceed only with failover, even
if the feature is called "loadbalancing". The value of "loadbalancing"
determine the order in which the failover occur on the configuration
(in serie or random)(1)(2).
In order to avoid trailing connections to down ODBC instances, in
system_odbc.ini, configure the following :
- Set the KeepAlive parameter to 1 in order to
close any connection to a down server (3)(4)(5).
This will avoid the connections to be kept open to ODBC instances
which are down.
More, you can also configure 2 separated data sources and remove the
configuration of alternateservers and loadbalancing from the driver
configuration will make all the odbc connection going back to the
Primary Session Store. Note that from documentation, we support
failover only, and the driver loadbalancing configuration stands for
failover too.
Here we've removed the alternateservers and LoadBalancing
configuration and we've defined 2 separate Session Store data sources,
1 for the Primary and another one for the Secondary.
system_odbc.ini :
[Session Primary]
Driver=/opt/CA/siteminder/odbc/lib/NSora27.so
Description=DataDirect 7.1 Oracle Wire Protocol
HostName=duspa01-u161157.ca.com
PortNumber=1521
LoginID=SMUSER
Password=xxxxx
ServiceName=XE
#SID=XE
CatalogOptions=0
ProcedureRetResults=0
EnableDescribeParam=0
EnableStaticCursorsForLongData=0
ApplicationUsingThreads=1
DMCleanup=2
[Session Secondary]
Driver=/opt/CA/siteminder/odbc/lib/NSora27.so
Description=DataDirect 7.1 Oracle Wire Protocol
HostName=duspa01-u161156.ca.com
PortNumber=1521
LoginID=SMUSER
Password=xxxxx
ServiceName=XE
#SID=XE
CatalogOptions=0
ProcedureRetResults=0
EnableDescribeParam=0
EnableStaticCursorsForLongData=0
ApplicationUsingThreads=1
DMCleanup=2
In the SmConsole, we've configured the Session Store connection
putting each of the datasources, separated with a coma "," :
sm.registry :
HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\Database\SessionServer=235352101
Data Source=Session Primary, Session Secondary; REG_SZ
Enabled= 0x1; REG_DWORD
MaxConnections= 0x20; REG_DWORD
Password= {RC2}cJhuVJ9LzQf12y9hk6FdGc9ztDo+Y8Em; REG_SZ
ProviderNamespace= ODBC:; REG_SZ
Use Default= 0; REG_DWORD
User Name= SMUSER; REG_SZ
Once done, we've started Policy Server, and we've seen it using the
Primary Session Store. When the Primary Session Store went off line,
Policy Server moved connections to the Secondary Session Store. When
the Primary Session Store came back online, then Policy Server moved
back the connections to the Primary Session Store :
smps.log :
[13473/4151690960][Fri Dec 28 2018
03:08:08][CServer.cpp:4951][INFO][sm-Server-01430] Logging and Trace
output in local time. TimeZone: [GMT-5:00]. Daylight Savings: 0
[13473/3799419760][Fri Dec 28 2018
03:08:12][CSmDbSessionManager.cpp:585][INFO][sm-Server-04350] Using
ODBC 'Session Data' data source 'Session Primary'.
[13473/4151690960][Fri Dec 28 2018
03:08:12][SmSSInDBStore.cpp:299][INFO][sm_LoginLogout_02014]
Initialized Session Server. Schema version is 3, lVersion
[13473/3715414896][Fri Dec 28 2018
03:09:12][CSmDbSessionManager.cpp:180][INFO][sm-Server-04350] Using
ODBC 'Session Data' data source 'Session Primary'.
[13473/4065667952][Fri Dec 28 2018
03:17:38][CSmDbSessionManager.cpp:147][INFO][sm-Server-04330]
Failing over to 'Session Data' data source 'Session
Secondary'.
[13473/3715414896][Fri Dec 28 2018
03:22:12][CSmDbSessionManager.cpp:155][INFO][sm-Server-04340]
Failing back to 'Session Data' data source 'Session
Primary'.
Netstat output during the use case reproduction :
We start the Policy Server :
Fri Dec 28 03:08:13 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED
12171/xe_pmon_XE
tcp 0 0 10.162.31.240:31701 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
We put load and Policy Servers build connections to the Primary
Session Store (.240):
Fri Dec 28 03:16:07 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:59662 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:32757 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED
12171/xe_pmon_XE
tcp 0 0 10.162.31.240:20816 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:22829 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:31004 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 689 10.162.31.240:31701 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:65377 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
We shut down the Primary Session Server (.240) :
Fri Dec 28 03:17:37 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 1 0 10.162.31.240:47888 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
tcp 1 0 10.162.31.240:59662 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
tcp 0 0 10.162.31.240:44748 10.162.31.240:1521 ESTABLISHED
12171/xe_pmon_XE
tcp 1 0 10.162.31.240:20816 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
tcp 1 0 10.162.31.240:22829 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
tcp 1 0 10.162.31.240:31004 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
tcp 1 0 10.162.31.240:65377 10.162.31.240:1521 CLOSE_WAIT
13473/smpolicysrv
Policy Server fails over connections to the Secondary Session Store
(.106) :
Fri Dec 28 03:18:38 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
We start over the Primary Session Server :
Fri Dec 28 03:21:32 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED
18172/xe_pmon_XE
tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
Policy Server notices it and starts to move back all connections to
the Primary Session Store :
Fri Dec 28 03:21:41 EST 2018
tcp 0 0 10.162.31.240:11821 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED
18172/xe_pmon_XE
tcp 0 0 10.162.31.240:39907 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:17993 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:55709 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:55979 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:36960 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:51719 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:22171 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:27555 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:58619 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:48457 10.162.31.106:1521 ESTABLISHED
13473/smpolicysrv
Until all connections are back to the Primary Session Store :
Fri Dec 28 03:21:48 EST 2018
tcp 0 0 10.162.31.240:45040 10.162.31.240:1521 ESTABLISHED
18172/xe_pmon_XE
tcp 0 0 10.162.31.240:39907 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:14352 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:36902 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 679 10.162.31.240:55979 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:27776 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:51390 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:27555 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
tcp 0 0 10.162.31.240:31303 10.162.31.240:1521 ESTABLISHED
13473/smpolicysrv
As stated earlier in this case, you won't be able to achieve
loadbalancing between the Session Stores. So you will never get
connections equally distributed between nodes.
As our documentation states, Session Stores instances should be set as
failover (6).
Additional Information
(1)
Configure an Oracle Data Source for CA Single Sign-On
AlternateServers=
If the primary server is not accepting connections, specifies the
connection failover to the other Oracle nodes.
Example:
(HostName=nete_servername2:PortNumber=1521:ServiceName=nete_servicename[,...])
LoadBalancing=1
Turns on client load balancing, which helps to distribute new
connections to keep RAC nodes from being overwhelmed with
connection requests. When enabled, the order in which primary and
alternate database servers are accessed is random.
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/installing/install-a-policy-server/configure-odbc-databases-as-policy-session-key-and-audit-stores/configure-odbc-databases-as-session-store/store-session-information-in-oracle.html
(2)
Policy Server and ODBC Store inconsistent loadbalancing connections
Even if the nomenclature of the configuration suggest there's a
loadbalancing, the following configuration means failover, and
not loadbalancing (1).
AlternateServers=(HostName=myserver:PortNumber=1111:ServiceName=myserver.mydomain.com)
LoadBalancing=1
https://knowledge.broadcom.com/external/article?articleId=119929
(3)
Policy Server Oracle Server Wire Protocol configuration
KeepAlive=0
https://knowledge.broadcom.com/external/article?articleId=32766
(4)
TCP Keep Alive
KeepAlive (KA)
If set to 0 (Disabled), the driver does not enable TCPKeepAlive.
If set to 1 (Enabled), the driver enables TCPKeepAlive.
https://media.datadirect.com/download/docs/odbc/allodbc/index.html#page/odbc/tcp-keep-alive.html
(5)
Load Balance Timeout
Specifies the number of seconds to keep inactive connections
open in a connection pool. An inactive connection is a database
session that is not associated with an ODBC connection handle,
that is, a connection in the pool that is not in use by an
application.
https://media.datadirect.com/download/docs/odbc/allodbc/index.html#page/odbc%2Fload-balance-timeout.html%23
(6)
Policy Server to Session Store Communication
Allows for failover. If a primary session store fails, Policy Servers
failover to a secondary session store.
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/implementing/implementing-siteminder/architectural-use-cases.html
Feedback
