WS_REST that lasts more than 300 seconds Aborts with Request to manager: technical error

Products

• CA Automic Dollar Universe

Issue/Introduction

Executing a script/action longer than 5 minutes, to a certain end point, using the WS_REST uproc type aborts with the error below:

Request to manager: technical error.   Java Manager error.
An internal server error occured. Please check the logs. Connection reset

Increasing the Timeout in Java manager as below does not help, the server timeout (300s) applies and the request of the manager for webservices gets cancelled:.
Manager for Java -> Settings->Timeout for connection ( in seconds)B y default, the timeout is set to 5 minutes ( 300 sec). We tried to change it from the console,( update 3000 seconds. 30,000 etc.) but a superior value does not help.

A shorter script to the same endpoint executes and completes fine.
Executing the same using CURL completes fine for longer and shorter duration scripts.

The issue started to occur since some change in the servers (rest api endpoint and dollar universe node) timeout was performed to reduce to 300s the server timeout.

Environment

Dollar Universe 6.x and 7.x

Manager for Webservices - uprocs WS_REST

Cause

The problem is that the Manager for Java Webservices is based in Java and Java does not natively controls KeepAlive.
It uses the OS level TCP KeepAlive (the parameters that were changed).

The reason why the same endpoint invoked via cURL works is due to the fact that curl is not using OS TCP KeepAlive, it uses the TCP layer, which is different.

What our Manager is doing :

 Our client is actively waiting for a response via InputStream read.
 

From the OS's TCP layer's perspective, the connection is "in use", because the client app is actively reading.

So: No idle time = No TCP Keep-Alive probes sent.

The only way to manage it in JAVA is what we are already doing which is via a timeout.

Server timeout or TCP KeepAlive defined too small (5 minutes) on the involved servers and tcp keepalive not implemented in the Webservice REST endpoint.

Resolution

Change the server's timeout (that is closing the connection before the manager goes into timeout) AND/OR implement TCP KeepAlive on the REST API server side.