We have found the root cause of this problem, which is a bug in the Java version that DLP is using in version 10.5:
From sun Bug Database, there is a bug related to handling client hello extension which is the same with our problem. Please check http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6728126 for details.
In Java 1.6.0_12 release notes, this issue is fixed. Please check http://www.oracle.com/technetwork/java/javase/6u12-137788.html and search 6728126 (BugID).
In DLP version 11, we are using version 1.6.0_20 which contains the fix. We were able to verify that both openSSL tests (with and without ticket extension) work correctly with DLP v11.
We recommend the following:
1. Upgrade the systems to v11, which contains the Java fix and is working correctly.
2. As a temporary solution for v10.5, we suggest to explore with SendMail whether their product can be configured not to use the ticket extension. This option is working correctly with both v10.5 and v11 as was verified by our internal testing.