When Access Gateway (formerly Secure Proxy Server (SPS)) tries to do a SSL connection with the back end SharePoint application server, we see the error in the nohup logs as below: 

handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake 
http-bio-2001-exec-1, SEND TLSv1.2 ALERT: fatal, description = handshake_failure 
http-bio-2001-exec-1, WRITE: TLSv1.2 Alert, length = 2 

In sm trace:
[07/17/2018][19:16:58][5312][1520][2a35fd20-aad5b87a-14a85362-e2708bc0-add358d6-d8][Noodle::doGet][com.rsa.ssl.SSLException: Certificate for <<SERVER>/<IP ADDRESS>> is not trusted or bad certificate at com.netegrity.util.security.rsa.AbstractHostVerifier.verify(AbstractHostVerifier.java:119)]

Release:
Component: SMSPA

Enable Java Cryptography Extension (JCE) by:

1) Java 1.8.0_151 or later: 

In <jdk>/jre/lib/security/java.security uncomment:

#crypto.policy=unlimited

so it looks like:

crypto.policy=unlimited

or

2) Java version earlier than  1.8.0_151, by installing the correct JCE patch for your java version