SSL mutual authentication fail

Products

• CA API Gateway

Issue/Introduction

Inbound calls are failing with 401 Authentication Required errors in Layer7 10.1. The same call works fine in layer7 9.4 using the same certificates to authenticate the user on both versions. 

logs:- 

2023-05-18T12:18:10.131-0400 WARNING 805 com.l7tech.server.policy.assertion.identity.ServerAuthenticationAssertion: 4201: No credentials found!
2023-05-18T12:18:10.131-0400 WARNING 805 com.l7tech.server.message: Message was not processed: Authentication Required (401)

 

Environment

Release : 10.1

Cause

Java bug for some java versions does not support some signature algorithms (rsa_pss_*  algorithms).  

https://bugs.java.com/bugdatabase/view_bug?bug_id=8226374

https://bugs.openjdk.org/browse/JDK-8227445

Certificate hash code cannot been generated so it will crash when trying to verify the certificate. 

The tcpdump revealed that the problem is due to the client is picking rsa_pss_rsae_sh256 (0x8084) to use as signature algorithm which actually is not working in the java 1.8 for tls1.2. 

In the 9.4 instance, the client was using algorithm, rsa_pkcs1_shar256 (0x0401),  so it worked.  

Resolution

Workaround:  It is possible to work around the issue using jdk.tls.disabledAlgorithms by adding RSASSA-PSS to the string in the java.security file, this should be done on both sides (client & server).