Not be able to connect to gateway oauth manager from browser

Products

• CA API Gateway

Issue/Introduction

After upgrading to v11.1.2, we started getting reports from customers that Kerberos wasn't working. We are unable to connect to gateway OAuth Manager from the browser.. 

A tcpdump from client shows that Server Hello,  Certificate, Server Key Exchange packet length > 16K and client issue FIN to finish the connection after server handshake (handshake Protocol: Certificate Request). 

Environment

API Gateway 11.x

Cause

The certificate trust store has more certificates that could be used for signing the client.  

The Microsoft Schannel security provider has a size limit of 16 KB for the trusted certificate authorities list used during the TLS/SSL handshake. This limit applies to the list of root certificates and trusted issuers, which is used to verify the validity of a peer's certificate. 

Resolution

Review and update the certificates in the trusted store by removing the option 'Signing Client Certificates' for those certificates and also removed certificates that are no longer needed to make sure the certificate list sent during handshake is less than 16K.