Cannot connect over TLSv1.2 to Oracle Unified Directory store due to "Illegal parameter" error

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

We are configuring Oracle Unified Directory 11gR2 as User Directory over SSL (OUD is using TLSv1.2) and we are getting errors after setting the certificates. We can connect using an LDAP client and openssl as client and we have verified the certificate is correct, however when we try to see the contents through AdminUI, we are getting the following errors:

On smps.log:

[51425/140734375143168][Wed Sep 20 2017 15:07:58][SmDsLdapConnMgr.cpp:923][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server 10.0.0.1 : 2636. Error 81-Can't contact LDAP server 

On OUD error log:

[20/Sep/2017:15:06:57 +0200] CONNECT conn=1211987 from=10.0.0.1:59973 to=10.0.0.1:2636 protocol=LDAPS 
[20/Sep/2017:15:06:57 +0200] DISCONNECT conn=1211987 reason="I/O Error" msg="Received fatal alert: illegal_parameter" 

On network traces, we do see the Client Hello from the Policy Server, and the Server Hello showing the cipher suite: 51 -> TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033). Then, the illegal parameter error:

    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter) 
        Content Type: Alert (21) 
        Version: TLS 1.2 (0x0303) 
        Length: 2 
        Alert Message 
            Level: Fatal (2) 
            Description: Illegal Parameter (47) 

 

Environment

Policy Server : R12.7 on RHEL 7.3 OUD : 11gR2 on RHEL 6.6

Resolution

OUD 11gR2 uses JDK 7 version for encryption and Policy Server uses the NSS libraries as client. In R12.7, NSS 3.20 Basic ECC libraries are used, and supporting the TLS_DHE_RSA_WITH_AES_128_CBC_SHA cipher suite. For OUD to support it, it needs to ensure the LDAPS handler has enabled the JRE cipher suites, JDK has to be patched with JCE (Java Cryptography Extension), and JVM version have to be upgraded to 1.7.0_161 or higher.