Customer has upgraded target Solaris server to Solaris 11.3 and apply Oracle Critical Patch Update released on 1st June 2018. When he/she tried to verify password it failed with the following error message.
  "Unable to establish communication channel to remote host"
Target application setup is using UNIX target connector.
PAM tomcat log shows the following exception.

Jun 25, 2018 4:16:39 AM com.cloakware.cspm.server.app.impl.ll c
WARNING: **** ACCOUNT VERIFICATION FAILED: targetAccount ID: <nnnn>' due to 'Error Code: 15212
Error Details: null
Error Message: Failed to establish a communications channel to the remote host.
Exception: com.cloakware.cspm.server.plugin.NetConnectorException: Failed to establish a communications channel to the remote host.
Stack Trace: com.cloakware.cspm.server.plugin.NetConnectorException: Failed to establish a communications channel to the remote host.
       at com.cloakware.cspm.server.plugin.SSHConnector.connect(SSHConnector.java:135)
       at com.cloakware.cspm.server.plugin.ChannelBeanShellScriptProcessorImpl.getConnectedChannel(ChannelBeanShellScriptProcessorImpl.java:401)
       at com.cloakware.cspm.server.plugin.ChannelBeanShellScriptProcessorImpl.<init>(ChannelBeanShellScriptProcessorImpl.java:88)
       at com.cloakware.cspm.server.plugin.ChannelBeanShellScriptProcessorImpl.<init>(ChannelBeanShellScriptProcessorImpl.java:121)
       at com.cloakware.cspm.server.plugin.targetmanager.UnixAdvancedTargetManager.verifyCredentials(UnixAdvancedTargetManager.java:89)
       at com.cloakware.cspm.server.app.TargetManager.run(SourceFile:648)
Caused by: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Protocol error: no matching DH grp found
       at com.jcraft.jsch.Session.read(Session.java:996)
       at com.jcraft.jsch.Session.connect(Session.java:323)
       at com.jcraft.jsch.Session.connect(Session.java:183)
       at com.cloakware.cspm.server.plugin.SSHConnector.connect(SSHConnector.java:113)
       ... 5 more

On the Solaris server's /var/log/authlog the following message was shown

Jun 28 4:16:39 <hostname> sshd[23248]: [ID 800047 auth.info] Disconnecting: Protocol error: no matching DH grp found



 

The 1st June 2018 Oracle Critical Patch Update has upgraded SUN_SSH to SSH-2.0-Sun_SSH_2.4 and increased acceptable ssh key bit lengths. It doesn't accept 1023 bit lengths anymore which causes UNIX target connector's JSch module of PAM 2.8.x cannot connect.

PAM version 2.8.3.x, 2.8.4.x
Target Solaris Server version 11.3 with 1st June 2018 Critical Patch Update installed

There are 3 possible solutions:

1. PAM 3.1.x or 3.2 are not affected by this issue. Upgrade to PAM 3.1.2 or PAM 3.2.

2. If you run OpenSSH (OpenSSH_7.5p1) instead of default SUN_SSH on the Solaris server then you will not be affected by this issue. On the target Solaris 11.3 server switch from SUN_SSH to OpenSSH (Refer https://docs.oracle.com/cd/E53394_01/html/E54793/tsk-openssh.html#scrolltoc). Run the following command
     #sudo pkg set-mediator -I openssh ssh

3. Ideally, upgrading PAM is better solution as it will update UNIX target connector to utilize larger SSH keys. However, if that is not possible, the shorter primes in /etc/ssh/moduli on the target Solaris server need to be enabled for the older clients (uncomment the lines with the shorter values).
A number of lines in /etc/ssh/moduli look similar to this below.

#20170809065415 2 6 100 1023 5 <long hex number>
^ - remove the #

Do this for all the other lines with "1023" or "1535" in the 5th field and then restart ssh.
   #sudo svcadm -v restart ssh

Refer to online documentation about Verify Synchronized Target Account Passwords.