There are 3 possible solutions:
1. PAM 3.1.x or 3.2 are not affected by this issue. Upgrade to PAM 3.1.2 or PAM 3.2.
2. If you run OpenSSH (OpenSSH_7.5p1) instead of default SUN_SSH on the Solaris server then you will not be affected by this issue. On the target Solaris 11.3 server switch from SUN_SSH to OpenSSH (Refer https://docs.oracle.com/cd/E53394_01/html/E54793/tsk-openssh.html#scrolltoc
). Run the following command #sudo pkg set-mediator -I openssh ssh
3. Ideally, upgrading PAM is better solution as it will update UNIX target connector to utilize larger SSH keys. However, if that is not possible, the shorter primes in /etc/ssh/moduli
on the target Solaris server need to be enabled for the older clients (uncomment the lines with the shorter values).
A number of lines in /etc/ssh/moduli
look similar to this below.#20170809065415 2 6 100 1023 5 <long hex number>
^ - remove the #
Do this for all the other lines with "1023" or "1535" in the 5th field and then restart ssh. #sudo svcadm -v restart ssh