WebSphere Liberty Profile randomly stops accepting TCP/SSL connections.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

 Symptoms:

 The WebSphere Liberty Profile (WLP) randomly stops accepting TCP connections.

 The JVM may stop accepting incoming TCP connections for up to 30 minutes.

 During the problem time period, there are a lot of RUNNABLE threads in the following method, and there is nothing in  process even though the threads are reported as RUNNABLE.

...
at sun.security.pkcs11.wrapper.PKCS11.C_DestroyObject(Native Method)
at sun.security.pkcs11.SessionKeyRef.dispose(P11Key.java:1138)
at sun.security.pkcs11.SessionKeyRef.drainRefQueueBounded(P11Key.java:1114)
at sun.security.pkcs11.SessionKeyRef. (P11Key.java:1129)
at sun.security.pkcs11.P11Key. (P11Key.java:119)
at sun.security.pkcs11.P11Key$P11SecretKey. (P11Key.java:405)
at sun.security.pkcs11.P11Key.secretKey(P11Key.java:292)
at sun.security.pkcs11.P11SecretKeyFactory.createKey(P11SecretKeyFactory.java:267)
at sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:175)
at sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:111)

Environment:

• WebSphere Liberty Profile 8.5.5.7
• Java VM version "Java HotSpot(TM) 64-Bit Server VM 1.8.0_45" from Oracle Corporation
• Solaris 10 Sparc 
• APM version  9.7.1.16 (Build 16)

Resolution:

The behavior has been reported with the RUNNABLE threads that resembles this Solaris 10 bug:

No response in calling sun.security.pkcs11.wrapper.PKCS11.C_DestroyObject(Native

To ignore this Java security provider, the customer tested disabling the SunPKCS11 crytographic provider as below in system and this solved the problem.
-Dsun.security.pkcs11.enable-solaris=false


Notes:
The " sun.security.pkcs11.enable-solaris=false", is actually a Java problem and nothing related to APM. Excluding it should mean Java uses some other public key encryption standards from the security file ($JAVA_HOME/lib/security/java.security)

Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component: