The size of the handshake message (40520) exceeds the maximum allowed size (32768)

Products

• CA API Gateway

Issue/Introduction

After the upgrade to API Gateway 11.1, gateway service does not start and the following message is seen in controller logs (sspc.log):

2023-09-26T09:19:45.751+0200 WARNING 1 com.l7tech.server.processcontroller.o: default may still be starting, but API is throwing unexpected exceptions
javax.xml.ws.WebServiceException: Could not send Message.
    at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:183)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
    at com.sun.proxy.$Proxy97.ping(Unknown Source)
    at com.l7tech.server.processcontroller.o.b(Unknown Source)
    at com.l7tech.server.processcontroller.o.a(Unknown Source)
    at com.l7tech.server.processcontroller.ProcessController.a(Unknown Source)
    at com.l7tech.server.processcontroller.ProcessController.a(Unknown Source)
    at com.l7tech.server.processcontroller.ProcessControllerDaemon.a(Unknown Source)
    at com.l7tech.server.processcontroller.ProcessControllerDaemon.main(Unknown Source)
Caused by: javax.net.ssl.SSLProtocolException: SSLProtocolException invoking https://localhost:2124/ssg/services/processControllerNodeApi: The size of the handshake message (40520) exceeds the maximum allowed size (32768)

--------------------

Caused by: javax.net.ssl.SSLProtocolException: The size of the handshake message (40520) exceeds the maximum allowed size (32768)

Environment

Release : API Gateway v11.0

Debian Linux

Cause

Change was introduced in the JDK long time ago with a limit on the size of the TLS handshake messages to 32768 (default value).
That affect customers who have a lot of certificates with  "signing client certificates" enabled.

Resolution

Set the maxHandshakeMessageSize=65536 for ProcessController.

# cd /etc/systemd/system

# chmod u+wx ssg.service

# nano ssg.service

Add / modify

Environment="PC_JAVAOPT=${ADD_OPENS} -Djava.security.egd=file:/dev/./urandom -Djdk.tls.maxHandshakeMessageSize=65536"

Save

# reboot -n

# ps aux | grep java | grep tls
layer7     950 13.2  4.2 5215480 256688 ?      Ssl  11:02   0:12 /opt/SecureSpan/JDK/bin/java --add-opens=jdk.crypto.ec/sun.security.ec=ALL-UNNAMED -Djava.security.egd=file:/dev/./urandom -Djdk.tls.maxHandshakeMessageSize=65536 -jar Controller.jar