JWP/JCP will not start, unable to connect, U00003590 UCUDB - DB error: '08006', 'IO Error: Connection reset', '17002', 'java.sql.SQLRecoverableException'

Products

• CA Automic One Automation

Issue/Introduction

After updating servers from powerbroker to centrify on the AE server, processes were restarted.  AWI showed that RestEndpoints were not available which points to the JCP and/or JWP being unavailable.  When attempting to start them, the following message occurs in the logs sometimes:

20210731/160914.492 - 1      U00003545 UCUDB: Opening database ...
20210731/161119.975 - 1      U00003611 DB OPEN executed. Return Code = '17002'
20210731/161119.975 - 1      U00003590 UCUDB - DB error: '08006', 'IO Error: Connection reset', '17002', 'java.sql.SQLRecoverableException'
20210731/161119.976 - 1      U00032031 Error when connecting to Database
20210731/161119.998 - 54     U00003432 Termination of Server 'AE_PRD#CP' initiated.
20210731/161120.004 - 54     U00003410 Server 'AE_PRD#CP' version '12.3.3+hf.3.build.1596133114839' ended abnormally.

Also attempting to run the ucsrvjp -setup throws the following, again, sometimes:

JCP and JWP will not start from service manager, but will start sometimes (anywhere from 1 in 5 attempts to 1 in 10 attempts) with nohup command.

Environment

Release : 12.3

Component : AUTOMATION ENGINE

Resolution

the error that seen in the JWP log:
U00003590 UCUDB - DB error: '08006', 'IO Error: Connection reset', '17002', 'java.sql.SQLRecoverableException'
is being returned by the jdbc driver when it is trying to connect to the Oracle database.  

In this case, the error is pointing to the connection being reset.  It does not specify what is being reset, just that it is happening.  This means that the connection from jdbc to the database is made, but that something is closing it.  A DBA can confirm this as well with the connection timing out.  So the jdbc driver is able to connect, but not stay connected.  This could be caused by a firewall, port scanner, network problem, etc...

Confirming the jdbc driver to use:
It would be a good idea to rule out any minor version mismatch between the jdbc driver and the database by using the ojdbc*.jar file that is in the $ORACLE_HOME/jdbc/lib directory.  Again, a DBA should be able to find and point to this file.

How to test the jdbc driver:

• There are a number of tools that can be used that are 3rd party.  One can be found here: https://github.com/aimtiaz11/oracle-jdbc-tester  but it's important to note that you cannot define which jdbc driver to use, it is delivered with the tester tool itself.  Using this would show (or rule out) issues with the jdbc driver itself if the connection times out with this as well.
• Oracle should also deliver a jdbc tester, there used to be one delivered in the samples folder of the oracle delivery; your DBAs may know more about where to find one.
• The best way to test will be using the same jdbc driver that is used in the automationengine /bin/lib directory as this is the one that the JWP is using to connect to the database, so whatever is used in the test should also be used by the JWP.

Other admin teams need to be heavily involved to troubleshoot the full stack to see what exactly is happening between JDBC and the database