SSH Connections Through PAM Hang When Using Stronger Kex Algorithms

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

After configuring a Linux target server to use stronger kex algorithms, SSH connections through Privileged Access Manager (PAM) fail.

When attempting a Putty connection through PAM, the following error occurs.

Remote side sent disconnect message
type 7 (service not available):
"Cannot establish SSH TCP service. Please contact the administrator.(code:5)"

When using the SSH applet, the connection appears to hang. The window shows as connected, but the command line never appears, nor does any error.

Cause

Looking at /var/log/secure on the target server, the following error occurred when PAM was trying to connect.
sshd[PID]: fatal: kexecdh_server: EC_KEY_new_by_curve_name failed

In this case, openSSH accepted the connection from PAM but could not complete the connection due to openSSL not being properly configured, resulting in the PAM SSH applet sitting idle.

Resolution

Any ecdh kex algorithm listed in sshd_conf should have a match in the output of the command `openssl ecparam -list_curves`. If not, either remove the kex algorithm list in sshd_conf or add the matching curve to openSSL.