javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: End user tried to act as a CA
search cancel

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: End user tried to act as a CA

book

Article ID: 262947

calendar_today

Updated On:

Products

SITEMINDER CA BCS Premier for CA Single Sign-On CA Single Sign On Agents (SiteMinder) CA Single Sign On Federation (SiteMinder) CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On

Issue/Introduction

Customer recently changed their CA Root certificates (from DigCert to Entrust) used for back end application.

After the change, one of the back application works fine, but another back end application proxy does not work.

User gets Noodle exception over the browser, and in agent trace log, shows error:

03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Retrying to send the request to backend web server.Retry count: 1]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Sending request to backend = backend.app.net url = https://backend.app.net/favicon.ico]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][requestConnection(): ][Get connection: {s}->https://backend.app.net:443, timeout = 900000]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][openConnection()][Connecting to backend.app.net/x.x.x.x:443]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][releaseConnection(): ][Released connection is not reusable.]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: End user tried to act as a CA]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Retrying to send the request to backend web server.Retry count: 2]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Sending request to backend = backend.app.net url = https://backend.app.net/favicon.ico]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][requestConnection(): ][Get connection: {s}->https://backend.app.net:443, timeout = 900000]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][openConnection()][Connecting to backend.app.net/x.x.x.x:443]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][releaseConnection(): ][Released connection is not reusable.]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: End user tried to act as aCA]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Retrying to send the request to backend web server.Retry count: 3]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][execute][Tried to send the request to backend web server three times.Throwing the exception to client. ]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][Noodle::doGet][com.ca.sso.smssl.SMSSLException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: End user tried to act as a CA at com.ca.sso.smssl.socket.SMSSLSocketImpl.startHandshake(SMSSLSocketImpl.java:403)]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][ErrorPageImpl::displayMessage][Custom Error Pages : Custom message is not an URL. If URL is specified then it mightnot be in proper format. Considering it as plain text message.]
[03/28/2023][13:16:11][559][140142457829120][ab961e65-1da67b67-f73ec771-b44d4569-4094b255-85][ProxyValve::invoke][Leaving the agent.]

Environment

Release : 12.8.03

Cause

Have verified ca-bundle.cert was updated on both Access gateways with new rootCA cert chain from Entrust.

External web sites indicate missing Basic Contraints extension within the certificate itself could be the possible root cause, but that is not the issue here either.

After compared the certificates chains from DigCert to Entrust, no major difference. Basic Contraints extension are present.

  • In intermediate cert:
Subject Type=CA
Path Length Constraint=0
  • In RootCA cert:
Subject Type=CA
Path Length Constraint=None.
 

When testing SSL connection using below command to get new root certificate, unfortunately, the response certificate chain was broken.

openssl s_client -connect backend.app.net:443 -showcerts 

Access gateway never received the complete root cert chain as it supposes to.  Similar command to other back end server does return cert chain properly. 

Since Access gateway did not receive the rootCA cert chain at all,  the error "End user tried to act as a CA" occurs, aka, missing "Basic Contraints extension within the certificate".  

Resolution

The backend application didn't import the intermediate certificate when they were importing into their Application Load Balancer's.

After Application Load Balancer reimported the required certs and then it works as expected. 

Additional Information

https://www.ibm.com/support/pages/end-user-tried-act-ca

http://www.mqseries.net/phpBB2/viewtopic.php?t=49364

https://knowledge.broadcom.com/external/article?articleId=241326