There are 3 possible solutions:
1. PAM 3.1.x or 3.2 are not affected by this issue. Upgrade to PAM 3.1.2 or PAM 3.2.
2. If you run OpenSSH (OpenSSH_7.5p1) instead of default SUN_SSH on the Solaris server then you will not be affected by this issue. On the target Solaris 11.3 server switch from SUN_SSH to OpenSSH (Refer
https://docs.oracle.com/cd/E53394_01/html/E54793/tsk-openssh.html#scrolltoc). Run the following command
#sudo pkg set-mediator -I openssh ssh3. Ideally, upgrading PAM is better solution as it will update UNIX target connector to utilize larger SSH keys. However, if that is not possible, the shorter primes in
/etc/ssh/moduli on the target Solaris server need to be enabled for the older clients (uncomment the lines with the shorter values).
A number of lines in
/etc/ssh/moduli look similar to this below.
#20170809065415 2 6 100 1023 5 <long hex number>^ - remove the #
Do this for all the other lines with "1023" or "1535" in the 5th field and then restart ssh.
#sudo svcadm -v restart ssh