Getting AwE-9999 errors and the details show ‘failed attempt at closing bad connection’ message.
Article ID: 87855
Updated On:
Products
CA Automic Applications Manager (AM)
Issue/Introduction
Error Message :
ErrorMsg: AwE-9999 Internal error (7/13/16 11:36 AM)
Details: failed attempt at closing bad connection
Jobs may go into a LAUNCH ERROR status or Applications Manager may seem to be hung or not able to reconnect to the database and in the RmiServer debug log the above error message is displayed.
This message indicates that Applications Manager encountered an error and was unable to re-connect. Usually, there will be additional errors in conjunction with the ‘failed attempt at closing bad connection’ that will help to determine the cause. However, in order to see those errors you will need to enable RmiServer debugging.
Once you have RmiServer debug logs, look for additional Oracle errors by searching for the following strings:
The additional information that is provided with debug enabled will help to narrow down what is causing the error.
Below are examples of the type of additional errors that may be seen when searching the debug logs using the above strings.
Example 1.
12:15:34.668 awserverWorker4: TRN:411: .DBAccess: checkError 17008
12:15:34.669 awserverWorker4: TRN:411: AwE-5001
12:15:34.669 awserverWorker4: TRN:411: SQL Error: 17008
12:15:34.669 connCkr: AwE-9999
ErrorMsg: AwE-9999 Internal error (7/13/16 12:15 PM)
Details: jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = alptrnsdb001.trans.ge.com) (PORT = 1521)) (ADDRESS = (PROTOCOL = TCP) (HOST=alptrnsdb002.trans.ge.com) (PORT = 1521)) (ADDRESS = (PROTOCOL = TCP) (HOST = alptrnsdb003.trans.ge.com) (PORT = 1521)) (LOAD_BALANCE = FALSE) (FAILOVER = TRUE) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = tsawstg1) ) ):AUTOMIC connection found to be is use for more than 5 minutes and validate method failed, removing connection from pool
ErrorMsg: AwE-5001 Database Query Error (7/13/16 12:15 PM)
Details: getConnection
null
java.sql.SQLException: shutdown
In the above example, there was a JDBC error ORA-17008 encountered which means the connection was closed and the reason provided in the logs was that the connection was found to be in use for more than 5 minutes and the validate method failed and was removed from the connection pool.
In order to determine why the connection was closed, additional research would need to be conducted by the DBA or the System Administrator. One reason that this error may be seen is when the ‘Keep-Alive’ settings on the server and/or database are set too high, for example to 20 minutes, so that could be one thing to check.
Example 2.
If Applications Manager seems hung or is unable to reconnect, and while searching for the same strings as above in the RmiServer debug there are messages similar to these:
java.sql.SQLException: Io exception: Software caused connection abort: recv failed
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
This indicates that the operation couldn’t be completed within the time out period which caused the connection to fail. And in the errors that follow the ORA-12560: TNS:protocol adapter error and the TNS-00505: Operation timed out errors indicates that there is a problem connecting to Oracle. This could be caused for several reasons such as if the database is behind a firewall and the firewall has an ‘idle session timeout’ value set and if the connection is idle for more that the defined period of time, it will drop the connection. Contact your DBA to troubleshoot the issue.
Example 3.
A long running subvar may also cause a connection from AM to the database to be lost. In this case, errors similar to those below will be found in the log after the ‘Details: failed attempt at closing bad connection’ message.
Error evaluating subvar #rms_threadnum for job: 5298008
03:15:21.936 SJ4: AwE-9999
03:15:21.936 SJ4: SQL Error: 2068
ErrorMsg: AwE-9999 Internal error (12/27/15 3:15 AM)
Details: 5298008
java.sql.SQLException: ORA-02068: following severe error from RETEK_LINK_RMS
ORA-03135: connection lost contact
An ORA-02068 error indicated a severe error received from the database link while processing jobid 5298008. Contact your DBA or System Administrator to troubleshoot the issue.
If you need additional assistance to resolve the issue, contact Automic Support.
ErrorMsg: AwE-9999 Internal error (7/13/16 11:36 AM)
Details: failed attempt at closing bad connection
Jobs may go into a LAUNCH ERROR status or Applications Manager may seem to be hung or not able to reconnect to the database and in the RmiServer debug log the above error message is displayed.
This message indicates that Applications Manager encountered an error and was unable to re-connect. Usually, there will be additional errors in conjunction with the ‘failed attempt at closing bad connection’ that will help to determine the cause. However, in order to see those errors you will need to enable RmiServer debugging.
Once you have RmiServer debug logs, look for additional Oracle errors by searching for the following strings:
‘.DBAccess: checkError’
‘AwE-‘
‘Details: ‘
'ORA-'
‘AwE-‘
‘Details: ‘
'ORA-'
The additional information that is provided with debug enabled will help to narrow down what is causing the error.
Below are examples of the type of additional errors that may be seen when searching the debug logs using the above strings.
Example 1.
12:15:34.668 awserverWorker4: TRN:411: .DBAccess: checkError 17008
12:15:34.669 awserverWorker4: TRN:411: AwE-5001
12:15:34.669 awserverWorker4: TRN:411: SQL Error: 17008
12:15:34.669 connCkr: AwE-9999
ErrorMsg: AwE-9999 Internal error (7/13/16 12:15 PM)
Details: jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = alptrnsdb001.trans.ge.com) (PORT = 1521)) (ADDRESS = (PROTOCOL = TCP) (HOST=alptrnsdb002.trans.ge.com) (PORT = 1521)) (ADDRESS = (PROTOCOL = TCP) (HOST = alptrnsdb003.trans.ge.com) (PORT = 1521)) (LOAD_BALANCE = FALSE) (FAILOVER = TRUE) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = tsawstg1) ) ):AUTOMIC connection found to be is use for more than 5 minutes and validate method failed, removing connection from pool
ErrorMsg: AwE-5001 Database Query Error (7/13/16 12:15 PM)
Details: getConnection
null
java.sql.SQLException: shutdown
In the above example, there was a JDBC error ORA-17008 encountered which means the connection was closed and the reason provided in the logs was that the connection was found to be in use for more than 5 minutes and the validate method failed and was removed from the connection pool.
In order to determine why the connection was closed, additional research would need to be conducted by the DBA or the System Administrator. One reason that this error may be seen is when the ‘Keep-Alive’ settings on the server and/or database are set too high, for example to 20 minutes, so that could be one thing to check.
Example 2.
If Applications Manager seems hung or is unable to reconnect, and while searching for the same strings as above in the RmiServer debug there are messages similar to these:
java.sql.SQLException: Io exception: Software caused connection abort: recv failed
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
This indicates that the operation couldn’t be completed within the time out period which caused the connection to fail. And in the errors that follow the ORA-12560: TNS:protocol adapter error and the TNS-00505: Operation timed out errors indicates that there is a problem connecting to Oracle. This could be caused for several reasons such as if the database is behind a firewall and the firewall has an ‘idle session timeout’ value set and if the connection is idle for more that the defined period of time, it will drop the connection. Contact your DBA to troubleshoot the issue.
Example 3.
A long running subvar may also cause a connection from AM to the database to be lost. In this case, errors similar to those below will be found in the log after the ‘Details: failed attempt at closing bad connection’ message.
Error evaluating subvar #rms_threadnum for job: 5298008
03:15:21.936 SJ4: AwE-9999
03:15:21.936 SJ4: SQL Error: 2068
ErrorMsg: AwE-9999 Internal error (12/27/15 3:15 AM)
Details: 5298008
java.sql.SQLException: ORA-02068: following severe error from RETEK_LINK_RMS
ORA-03135: connection lost contact
An ORA-02068 error indicated a severe error received from the database link while processing jobid 5298008. Contact your DBA or System Administrator to troubleshoot the issue.
If you need additional assistance to resolve the issue, contact Automic Support.
Environment
Release: AAMOS499000-8.0-Automic Applications Manager-OS400 Agent
Component:
Component:
Cause
Cause type:
By design
Root Cause: Refer to description.
By design
Root Cause: Refer to description.
Resolution
This field was added on 30/03/2017. This article has not been updated yet. Refer to the "Description" or "Workaround" sections for solution information.
Additional Information
Workaround :
N/A
N/A
Feedback
Yes
No
Powered by
