PORTAL API - HTTP ERROR

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

Today we had a PROD outage for about 3 hours when the portal encountered the below error. These messages were on the TDMWeb log file. I have attached all the logs for the day. Can you please tell us what went wrong and what is required to avoid this in the future?

 

2021-04-19 13:47:18.152 EST [ERROR] [https-openssl-nio-443-exec-168] --- [U:U82ABH][M:POST][P:/api/ca/v1/connectionProfiles/search/SearchProfile]              c.c.t.c.e.h.GlobalExceptionConvertor:  HTTP Error - INTERNAL SERVER ERROR: Exception: java.io.IOException: An established connection was aborted by the software in your host machine
java.io.IOException: An established connection was aborted by the software in your host machine

 

2021-04-19 13:47:18.153 EST [WARN ] [https-openssl-nio-443-exec-168] --- [U:U82ABH][M:POST][P:/api/ca/v1/connectionProfiles/search/SearchProfile]   o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver:  Failure in @ExceptionHandler org.springframework.http.ResponseEntity<?> com.ca.tdm.common.exception.handler.GlobalExceptionConvertor.handleAllOtherErrors(java.lang.Exception) throws java.io.IOException
java.io.IOException: An established connection was aborted by the software in your host machine

Environment

Release : 4.9

Component : CA Agile Requirements Designer - TMX Script Generator

Resolution

I strongly believe that tomcat was overloaded and requests were being ignored resulting in the error messages being displayed from TDMWeb.

By default, Tomcat sets maxThreads to 200, which represents the maximum number of threads allowed to run at any given time.

 

It is possible to change this default value to let's say 300 by adding it in tomcat/conf/server.xml. However you should be aware that asking for more resources means the system will be under more strain but hopefully tomcat will be able to manage the load

 

On my machine I have...

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLSv1.2" keystoreFile="${tdmweb.keystorePath}" keystorePass="${tdmweb.keystorePassword}" keyAlias="${tdmweb.keyAlias}" ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256" connectionTimeout="60000" maxHttpHeaderSize="3000000"/>

 

the new entry would be 

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLSv1.2" keystoreFile="${tdmweb.keystorePath}" keystorePass="${tdmweb.keystorePassword}" keyAlias="${tdmweb.keyAlias}" ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256" connectionTimeout="60000" maxHttpHeaderSize="3000000" maxThreads="300"/>

 

see https://tomcat.apache.org/tomcat-9.0-doc/config/http.html for more details.

 

We have both our PROD servers set to the 300 Max thread value and we haven’t seen this error in the past week.