Decrypting and decoding SSL/TLS-encrypted traffic with Wireshark

Products

• CA API Gateway

Issue/Introduction

Wireshark can be used to decode and decrypt SSL-TLS-encrypted communications between a client application and the CA API Gateway appliance. This article has the following limitations:

1. The Gateway is acting as the server in a TCP connection
2. The Gateway is not using a cipher suite based upon Diffie-Hellman key exchange.
3. A hardware security module is not in use with the Gateway appliance.

Decrypting SSL/TLS-encrypted traffic requires access to the private key used by the server. If the Gateway is the server for a TCP connection then the Gateway's private key can be exported and used. If the Gateway is a client for a TCP connection then it would be necessary to procure the key from the server or service administrator. This article will focus on using the Gateway as a server.

A packet capture cannot be decrypted if an SSL/TLS channel is opened with cipher suites using Diffie-Hellman key exchange (which includes elliptic curve ciphers). Diffie-Hellman key exchange allows for perfect forward secrecy. Perfect forward secrecy prevents an attacker from taking a packet capture and decrypting the capture later after a set of keys are compromised. This limitation prevents even a valid administrator from decrypting a packet capture after the transaction is complete.

Using a hardware security module prevents a packet capture from being decrypted as private keys present in the HSM cannot be exported. Private keys that were created elsewhere and stored within an HSM-secured keystore can still be used but cannot be exported from the Gateway and will have to be exported from another system.

Resolution

Table of Contents

• Exporting the necessary private key
• Configuring Wireshark to use the private key
• Verifying a TCP stream can be decrypted
• Troubleshooting a failed decryption

Exporting the necessary private key

1. Log in to the Policy Manager as an administrative user
2. Open the Manage Listen Ports task
3. Open the properties for the desired listen port
4. Select the SSL/TLS Settings tab
5. Verify the alias of the private key assigned to that port
6. Close all dialogs and open the Manage Private Keys task
7. Select the desired private key and select the Properties button
8. Select the Export Key button
9. Specify a passphrase and save the value for use later
10. Save the key to the workstation
11. Close all open dialogs

Configuring Wireshark to use the private key

1. Open Wireshark
2. Select the Preferences from the Edit menu
3. Select HTTP from the Protocols menu
4. Add the HTTPS port used to the SSL/TLS Ports field
5. Select SSL from the Protocols menu
6. Select the Edit button
7. Select the New button
8. Specify the IP address of the server
9. Specify the Port used to communicate with the server
10. Set the Protocol as http
11. Set the Key File to the PKCS#12 file exported from the Policy Manager
12. Specify the Password set when exporting the key from the Policy Manager
13. Select OK
14. Specify an "SSL debug file" by pointing to a text file. This text file will be created if it does not exist

Verifying a TCP stream can be decrypted

1. Open a new capture file in Wireshark
2. Specify the following Capture Filter: ssl.handshake
3. Find the Client Hello from the client IP address
4. Right-click the frame and select Follow SSL Stream
5. An HTTP transaction should be visible in clear text.

Troubleshooting a failed decryption

The SSL debug log specified previously will contain data for each packet dissection and decryption. Note the frame number (specified by the No. column) and open the SSL debug log. Search for this frame number (or a similar frame number) in this log and note the error message.