search cancel

High api response time

book

Article ID: 253922

calendar_today

Updated On:

Products

CA API Gateway

Issue/Introduction

We are getting a high response time on some API's.

OTK and JWT transaction long transaction time according to Dynatrace, the API gateway spend an amount of time on internal processing, and it impacts on total API response time.

Environment

Release : 9.2

Cause

BLOCK threads 

"tomcat-exec-executor-732" #167547 daemon prio=5 os_prio=0 tid=0x00007f32b820c000 nid=0xcc7e waiting for monitor entry [0x00007f3355d7c000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at com.l7tech.server.security.keystore.software.DatabasePkcs12SsgKeyStore.keyStore(Unknown Source)

- waiting to lock <0x00007f338e677da0> (a com.l7tech.server.security.keystore.software.DatabasePkcs12SsgKeyStore)

at com.l7tech.server.security.keystore.JdkKeyStoreBackedSsgKeyStore.getCertificateChain(Unknown Source)

at com.l7tech.server.security.keystore.SsgKeyStoreManagerImpl.lookupKeyByKeyAlias(Unknown Source)

at sun.reflect.GeneratedMethodAccessor645.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy103.lookupKeyByKeyAlias(Unknown Source)

at com.l7tech.server.DefaultKeyImpl.lookupKeyByKeyAlias(Unknown Source)

at com.l7tech.external.assertions.jwt.JwtUtils.getKeyFromStore(Unknown Source)

at com.l7tech.external.assertions.jwt.server.ServerDecodeJsonWebTokenAssertion.checkRequest(Unknown Source)

 

Resolution

The keystore file can get large.  Reading the keystore file from the database is very expensive.  More time spent on database reading keystore file means those waiting threads are hold for a longer period of time.  This affects performance.

Now we are looking at the method keystore in DatabasePkcs12SsgKeyStore, which has already implemented caching keystorefile mechanism by retrieving keystore file first and then comparing cached keystore file version and database version.  So, when the refreshing keystore period is coming, we can retrieve the keystore file version from the database instead of the actual keystore file, because the version is a very small integer compared to the larger keystore file.

DE353852  Resolution to apply the appropriate patch :

  1. Add caching for private keys and only Create/Decode/Encode Web Token assertions use that. 
    • 9.2 CR9
    • 9.3 CR3
  2. Flip logic for charset encoding resolution, use the faster nio library then fallback to the slower javax.mail library
    • 9.2 CR9
    • 9.3 CR3 
    • 9.4+ upgraded the javax.mail library which fixed this issue.