When Access Gateway (formerly Secure Proxy Server (SPS)) tries to do a SSL connection with the back end SharePoint application server, we see the error in the nohup logs as below:
handling exception:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
http-bio-2001-exec-1, SEND TLSv1.2 ALERT: fatal, description = handshake_failure
http-bio-2001-exec-1, WRITE: TLSv1.2 Alert, length = 2
In sm trace:
[07/17/2018][19:16:58][5312][1520][2a35fd20-aad5b87a-14a85362-e2708bc0-add358d6-d8][Noodle::doGet][com.rsa.ssl.SSLException: Certificate for <<SERVER>/<IP ADDRESS>> is not trusted or bad certificate at com.netegrity.util.security.rsa.AbstractHostVerifier.verify(AbstractHostVerifier.java:119)]