Client suddenly stops sending client certificate to gateway

Issue/Introduction:

HI Team, We are trying to hit one of our API and we are getting error as "Assertion Falsified" with 500 error - it is protected by mutual client SSL authentication and is no longer working.

Cause:

A new CA Certificate had been added, and it had the attribute "Sign Client" set.

With that attribute set, then subsequent SSL handshakes with the gateway were sending that DN out to the client as part of the ServerHello. Most client software (not all by a long shot) use that DN to restrict which client certificates can be selected/used to be sent back to the Gateway.

Previously this DN list had been empty, and so any client certificate could be selected, once this "Sign Client" CA Cert has been added the client cert was not being sent back from (some) client programs (RestMan for example, and Gateway Policy Manager ignore the recommendation and send back the client cert anyway).

Environment:

API Gateway 9.4

Resolution:

Adding the CA Cert that issued the client certificate and selecting "Sign Client" that that certificate enabled the client to work as before.

An alternative would be to uncheck the "Sign Client" on the newly added certificate or removing the new cert.

Additional Information:

Wireshark and also java setting "-Djava.net.debug=all" where used to view the SSL/TLS handshake.

The deal with the newly added CA cert, is that it has "Sign Client" enabled - prior to that they did not have any certs with "Sign Client" enabled.

With that one cert with "Sign Client" enabled - the gateway sends the SSL ServerHello response back to the client with client-cert request part like this :

*** CertificateRequest
Cert Types: RSA, DSS, ECDSA
Cert Authorities:
<CN=*.app.prod.e1.dev.ca.com, O="Broadcom, Inc.", L=San Jose, ST=California, C=US>

Previously (with no certs enabled for "Sign Client") the request looked like this:
*** CertificateRequest
Cert Types: RSA, DSS, ECDSA
Cert Authorities:
<Empty List>

And for the fix (for one app anyway) we added the issuer of their cert -so have two certs marked as "Sign Client" and they get :

Cert Types: RSA, DSS, ECDSACert Authorities:
<CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US>
<CN=*.app.prod.e1.dev.ca.com, O="Broadcom, Inc.", L=San Jose, ST=California, C=US>

Now, what the client program does with that client-cert-request part in the ServerHello is up to the client program - browsers will often use that list to determine which certificates are valid and pop up a list to let you choose one.
Some clients (RestMan, SSG Policy Manager) ignore it and just send whatever cert you selected to send with that URL.

They are using java SpringClient - which uses the normal java code, which uses that list to filter for valid certificates :

So :
a) When there is No DN names,

Cert Authorities:

<Empty>

The client seems it is happy to send any client cert (ie this is working - as per the original setup).

b) When there is just one not-matching cert :

With DN list :
Cert Authorities:

<CN=*.app.prod.e1.dev.ca.com, O="Broadcom, Inc.", L=San Jose, ST=California, C=US>

The Spring Client java responds with :

*** ServerHelloDone
[read] MD5 and SHA1 hashes: len = 4
0000: 0E 00 00 00 ....
Warning: no suitable certificate found - continuing without client authentication
*** Certificate chain
<Empty>

(ie failing)

c) When we put in two client certs as "Sign Client" :

Cert Authorities:
<CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US>
<CN=*.app.prod.e1.dev.ca.com, O="Broadcom, Inc.", L=San Jose, ST=California, C=US>

Then it works as normal because the client cert they are using is signed by DigiCert issuer - so matches.