Seeing SSL Decode failures with correct private keys and supported SSL cipher suites.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Symptoms:

The TIM SSL Server Status page shows a very high number of 'Connections with decode failures' (zero 'Unsupported cipher suites') for web server groups.

Note that the Web server SSL keys are current on TIM. Also APM CE (CEM) analysis still appears to be healthy i.e. the expected number of transactions are visible.

So are the high decode failures of concern?

<Please see attached file for image>

Environment:

This can impact all MTP/TIMs regardess of version.

Cause:

These symptoms can be caused when the TIM is seeing "false connections" to the web server but these packets contain no application data. This causes the TIM to generate a decode failure for each such connection. This previous KB article has the details: TEC609607 "Can SSL packet decode failures be due to TCP false connections?"  

Using Wireshark on a captured pcap file, it is possible to examine the number and length of conversations to further confirm that false connections are the root cause. e.g.

• Statistics->Conversation List ->TCP
• TCP conversations gives total number of conversations
• Order by number of Packets decreasing and can see that there are a huge number of conversations with a small number of packets.

<Please see attached file for image>

• Follow a TCP Stream for one of the low packet conversations and verify that the entire conversation has 0 bytes i.e. “Len=0” in both directions. Each such conversation would cause the TIM to generate a decode failure.

   

<Please see attached file for image>

 

The false connections are often from "health check" pings to the web servers from a Load Balancer. Therefore, the other side of the connection is often a common set of a few IPs.

After reviewing the above Conversations List to get the common IPs, then filter on those IPs to determine the actual number of small packet conversations (0 length) that those IPs are causing

e.g.

In this scenario 2 IPs were found to be generating the false connections and each one contributed 648 out of the total of 1554

<Please see attached file for image>

 

<Please see attached file for image>

 

So out 1554 total connections, 1296 came from the false connections i.e. ~ 84% 

Resolution:

CEM analysis is still healthy because the TIM is processing the connections that do contain application data, and the only really downside is extra overhead on the TIM. Try the following to stop the symptoms:

• If using a standard TIM, it would need to filter the traffic reaching the actual TIM to prevent sources of false connection type data.
• If using a MTP TIM, it can add a separate Hardware Filter similar to the default “HTTP - full packets” to EXCLUDE the common IPs generating the false connections.

Additional Information:

Is it possible to remove all SSL Decode failures? See:TEC1990782

Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component: