App service cannot be reached, Could not retrieve a valid tenant message thrown in logs

Products

• Clarity PPM On Premise

Issue/Introduction

Issue:

This issue occurs on a customer Oracle RAC issue, the loginTimeout in TenantConfigurationManager.java is dropping all the connections when they take beyond 5 sec. On some environments this is causing an issue : errors are thrown in the logs, the database servers are unavailable, systems are in outage.

Error messages that can be seen in the logs:

ERROR 2016-02-22 02:25:01,376 [http-nio-7020-exec-149] filter.RequestTenantIdExtractor (none:none:none:none)

com.niku.union.config.ConfigurationException: Could not retrieve a valid tenant.

at com.niku.union.config.TenantConfigurationManager.getValidLocalTenant(TenantConfigurationManager.java:355)

at com.niku.union.web.filter.RequestTenantIdExtractor.getTenantIdForRequest(RequestTenantIdExtractor.java:228)

at com.niku.union.web.filter.TenantSetupFilter.doFilter(TenantSetupFilter.java:49)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.niku.union.web.filter.CharsetFilter.doFilter(CharsetFilter.java:45) 

We also note random restarting of the affected services for app and bg.

Steps to Reproduce:

1. On a system with a slower connections to the database, install CA PPM 14.3
2. Note that the database connections do not get created when they hit 5 sec

Expected Result: App service cannot be reached, tenant is marked as invalid

Actual Result: App service should be reachable, tenant is still valid   

Cause:

Caused by CLRT-79927 

When troubleshooting with Oracle Support they indicated this is a timeout set on the client. 

Resolution: 

Resolved in CA PPM 14.3 Generic Patch #5   

Additional Information:

Reference TEC1696455 : PPM 14.3 Generic Patch available to resolve reported issues

Environment

Release: ESPCLA99000-14.3-Clarity-Extended Support Plus
Component: