Unable to Launch Client java.lang.OutOfMemoryError: Java heap space Protocol violation

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Error Message :
ErrorMsg: AwE-9999 Internal error 
Details: null
java.sql.SQLException: Protocol violation

ErrorMsg: AwE-9999 Internal error 
Details: null
java.lang.OutOfMemoryError: Java heap space

Users receive a connection refused error message when attempting to log into the Application Manager client.

A review of the $AW_HOME/log/RmiServer log files show the following errors.

MSM:read767-Agentname: .MasterSocketManager: cleanupSocket (sync) 3b353bbb[TLS_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=123.456.78.9,port=12120,localport=6101]]
ErrorMsg: AwE-9999 Internal error 
Details: null
java.sql.SQLException: Protocol violation
Followed by
OCh: null ErrorMsg: AwE-9999 Internal error 
Details: Java heap space
Null

This error message occurred when a user previously submitted a large history query from the GUI.

Investigation

This can be confirmed in the log files by analyzing the following sections of the RmiServer log.

A request is made to query job history form the gui

05:15:18.881 CR5 1.123.123.12:45489: .DBAccess: getSqlData() sID-12345678 APPWORX SELECT DISTINCT * from AW_JH_VIEW WHERE 1 = 1 AND (so_job_finished BETWEEN TO_DATE('2017-02-01 13:24:00','YYYY-MM-DDHH24:MI:SS') AND TO_DATE('2017-03-30 13:24:00','YYYY-MM-DD HH24:MI:SS'))order by so_job_finished desc

This launches the history request with the following sequence number (SeqNo 24228).

There is a large time gap in the logs containing the following error message.

CR51.123.123.12:45489: AwE-9999
CR51.123.123.12:45489: SQL Error: 17401
 MSM:read767-agentname: .MasterSocketManager: cleanupSocket (sync) 3b353bbb[TLS_DH_anon_WITH_AES_128_CBC_SHA: Socket[addr=automic.com/123.456.78.9,port=12120,localport=6101]]
ErrorMsg: AwE-9999 Internal error 
Details: null
java.sql.SQLException: Protocol violation
Followed by
OCh: null ErrorMsg: AwE-9999 Internal error 
Details: Java heap space
Null

Note that sequence number 24228 is being referenced in the error which indicates that the error is coming from the history query previously launched.

06:56:01.743 CR61.123.456.78.9:45489: .C: invoke getJobUpdates com.appworx.server.data.AxRmiServer
ErrorMsg: AwE-5103 network socket error 
Details: Response error SeqNo 24228 Agent 1.23.45.67:1099 Master Client service clientServices sessionID 1234 Method getHistoryQuery [com.appworx.shared.data.JobFilter@55821a8f, null]
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset

 
 

Environment

OS Version: N/A

Cause

Cause type:
By design
Root Cause: This issue is caused by hitting a java memory resource limit when submitting a large history query.

Resolution

Narrow the query results by:

1. limiting the initial query so fewer rows are returned

 OR

2. ​run the query outside of Applications Manager

Fix Status: Enhanced in version 9.4.4

Fix Version(s):
9.4.4

Additional Information

Starting in 9.4.4 the history query window now includes a 'Max Returned Rows' and 'Query Timeout' field. This was added in effort to help mitigate users from running large history queries from the client gui.