SSLProtocolException: handshake alert: unrecognized

Article Id: 140549

Status: Published

Updated On: 20-11-2019 20:19

Products:

CA Single Sign On Secure Proxy Server (SiteMinder) , AXIOMATICS POLICY SERVER , CA Single Sign On Agents (SiteMinder) , SITEMINDER , CA Single Sign On Federation (SiteMinder) , CA Single Sign On SOA Security Manager (SiteMinder) , SITEMINDER

Issue/Introduction:

Customer is getting javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name when accessing internal URL.

Environment:

Release : 12.8

Component : SITEMINDER -POLICY SERVER

Resolution:

Disable SNI:

1. Stop SPS/Access Gateway

2. Make a backup copy of CA\secure-proxy\proxy-engine\conf\SmSpsProxyEngine.properties

3. Edit CA\secure-proxy\proxy-engine\conf\SmSpsProxyEngine.properties and find:

NETE_SPS_PROXYENGINE_CMD="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Xms512m -Xmx1024m -XX:MaxMetaspaceSize=256M -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -DSM_AGENT_LOG_CONFIG="%STS_AGENT_LOG_CONFIG_FILE%" -Dfile.encoding=UTF8 -DIWACONFIGHOME="%IWACONFIGHOME%" -Dlogger.properties="%NETE_SPS_TOMCAT_HOME%\properties\logger.properties" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources;%NETE_SPS_ROOT%\agentframework\java\bc-fips-1.0.1.jar" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf"

4. Modify NETE_SPS_PROXYENGINE_CMD to add -Djsse.enableSNIExtension=false like the below.

NETE_SPS_PROXYENGINE_CMD="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Xms512m -Xmx1024m -XX:MaxMetaspaceSize=256M -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -DSM_AGENT_LOG_CONFIG="%STS_AGENT_LOG_CONFIG_FILE%" -Dfile.encoding=UTF8 -DIWACONFIGHOME="%IWACONFIGHOME%" -Dlogger.properties="%NETE_SPS_TOMCAT_HOME%\properties\logger.properties" -Djsse.enableSNIExtension=false -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources;%NETE_SPS_ROOT%\agentframework\java\bc-fips-1.0.1.jar" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf"

5. Start SPS/Access Gateway