PAM RDP client SSL connection error handshake_failure(40)

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

An attempt to launch an RDP session fails with error "SSL connection error handshake_failure(40).

 

Cause

The RDP server had a custom cipher suite list configured that was incompatible with the PAM RDP client. Command "nmap --script ssl-enum-ciphers -p 3389 <address>" returned the following list:

...

PORT     STATE SERVICE
3389/tcp open  ms-wbt-server
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp384r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp384r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|     compressors:
|       NULL

...

This list does not include any of the cipher suites documented on page Access Methods.

Resolution

Adding cipher suite TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 to the list resolved the problem. It required an RDP server reboot after updating the list.

Additional Information

See page TLS Cipher Suites in Windows Server 2022 and later, or similar pages covering older releases, for the list of default ciphers for a given Windows release and how to customize it using the group policy editor or powershell.