Rest functions are not thread safe

Products

• Clarity PPM On Premise

Issue/Introduction

The Clarity application has multiple threads trying to access a map concurrently causing race conditions. 

Application servers where rest calls are made sometimes experience corrupt tomcat servers. Those tomcat servers experience high cpu load (100% of total cpu) and generate nio threads that all end in an infinite loop in java.lang.TreeMap.put which is called from this class/method: com.ca.ppm.rest.resource.FunctionResourceFactory.registerFunction(FunctionResourceFactory.java:66).

A thread dump (attached) shows schema for all nio threads.

After inspection of the code in FunctionResourceFactory#registerFunction, it appears that TreeMap#put is called non-synchronized. All locked objects in the stacktrace are temporary objects. All stacktraced of the thread dumps have different object hashes for those locked objects. Thus TreeMap is not called thread safe. The Java API DOC of TreeMap explicitely stats that this implementation is not thread safe and should always be called wrapped in synchronized externally when used in a threaded environment (such as tomcat webservers).

This implies that the REST implementation of Clarity is not thread safe and rest calls may crash or corrupt tomcat servers. The only option is to kill the tomcat server.

 

Environment

Clarity 15.9.0 

Cause

This was reported as product defect DE59443

Resolution

Fixed in 15.9.1

This was identified and fixed with replacing tree map with ConcurrentSkipListMap 

Additional Information

https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/TreeMap.html

SSCCE that shows the issue: https://ivoanjo.me/blog/2018/07/21/writing-to-a-java-treemap-concurrently-can-lead-to-an-infinite-loop-during-reads/