Errors when trying to reconnect to Oracle DB

Products

• CA Cloud Test Mobile
• CA Application Test
• Service Virtualization

Issue/Introduction

We experience this issue when our DB teams are patching our database servers. Our best guess so far is that when the patching happens, the servers that house our DevTest DB is being restarted. After that restart the IAM is unable to reconnect to the DB which causes a bunch of errors. To resolve this, we usually end up restarting all DevTest components, and the DB connects just fine.

We are wondering if this is a known problem with DevTest? Also, what suggestions you might have for us to try to automatically reconnect to the DB after patching. 

 

Environment

All supported DevTest releases.

Cause

Oracle connection issue

Resolution

Seeing tons of these error messages in the IAM server.log file:

Caused by: java.sql.SQLRecoverableException: No more data to read from socket

Research on this error and found this information:

Answer: This error indicates an abort of a process, and the first step in resolving this error is to look in the alert log and look for a trace file. The "no more data to read from socket" is not a JDBC or an Oracle error. Rather, it is a generic communications error, often as the result of a TNS connectivity issue.

The "no more data to read from socket" error is most often because of an Oracle bug (in 11g) and we recommend moving to the latest release of Oracle.

My recommendation is before the next database patch schedule to do the following:

1. Bring down all of your DevTest components normally on all machines. (remember to shut down manually from Task Manager the extra java.exe process from IAM, this is fixed in 10.5.0)
2. Clean out the lisatmp folders to start with fresh log files.
3. Not sure if your services are set to Manual or Automatic, if not Automatic, start all in this order:
IAM, Enterprise Dashboard, Registry, Portal, VSE.
4. If this occurs again, after next patch, open a new support case and attach the new server.log, enterprisedashboard.log and registry.log files to the case.

The problem is happening with IAM and because that is not coming up properly, when the Enterprise Dashboard and Registry start they will get errors.