IJ000470: You are trying to use a connection factory that has been shut down

Products

CA Process Automation Base

Issue/Introduction

Encountering issues with CA Process Automation Manager (PAM) where the CA Process Automation Orchestrator Service fails to start. When we boot up the service, the process starts, but stops very shortly after boot up.

No configuration changes have been made.

The following are the last entries in the server.log after booting up the PAM server:

202X-09-13 06:38:17,651 ERROR [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0043: Deployment unit processor org.jboss.as.ee.beanvalidation.BeanValidationFactoryDeployer@7b92781 unexpectedly threw an exception during undeploy phase POST_MODULE of subdeployment "systemprops.jar" of deployment "c2oear-snapshot.ear": org.jboss.weld.exceptions.IllegalStateException: WELD-000229: Contextual reference of ValidatorFactoryBean [id=org.hibernate.validator.cdi.internal.ValidatorFactoryBean_default] is not valid after container c2oear-snapshot.ear shutdown

<omitted>

202X-09-13 06:38:17,651 ERROR [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0043: Deployment unit processor org.jboss.as.ee.beanvalidation.BeanValidationFactoryDeployer@7b92781 unexpectedly threw an exception during undeploy phase POST_MODULE of subdeployment "web-snapshot.war" of deployment "c2oear-snapshot.ear": org.jboss.weld.exceptions.IllegalStateException: WELD-000229: Contextual reference of ValidatorFactoryBean [id=org.hibernate.validator.cdi.internal.ValidatorFactoryBean_default] is not valid after container c2oear-snapshot.ear shutdown

<omitted>

202X-09-13 06:38:17,651 ERROR [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0043: Deployment unit processor org.jboss.as.ee.beanvalidation.BeanValidationFactoryDeployer@7b92781 unexpectedly threw an exception during undeploy phase POST_MODULE of deployment "c2oear-snapshot.ear": org.jboss.weld.exceptions.IllegalStateException: WELD-000229: Contextual reference of ValidatorFactoryBean [id=org.hibernate.validator.cdi.internal.ValidatorFactoryBean_default] is not valid after container c2oear-snapshot.ear shutdown

<omitted>

202X-09-13 06:38:18,244 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 51) WFLYSRV0022: Deploy of deployment "c2oear-snapshot.ear" was rolled back with no failure message

202X-09-13 06:38:18,307 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.

202X-09-13 06:38:18,323 ERROR [org.jgroups.protocols.JDBC_PING] (Thread-110) JGRP000115: Could not open connection to database: java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/RuntimeLibDS

<omitted>

202X-09-13 06:38:18,323 ERROR [org.jgroups.protocols.JDBC_PING] (Thread-110) JGRP000215: Failed to delete PingData in database

Environment

Release : 4.4

Cause

c2o.log Logging shows

202X-09-14 06:30:06,466 INFO  [com.optinuity.c2o.ddlgenerator.C2ODBUtil] [read Pool -- 97] Reading C2ODefinition XML at the path  C:\CA\PAM\wildfly\standalone\.c2orepository\.c2oserverresources\xml\C2ODBDefinition.xml
202X-09-14 06:30:06,497 INFO  [com.optinuity.c2o.ddlgenerator.C2ODDLGenerator] [read Pool -- 97] Starting Database adjustment**** ITPAMQUEUES
202X-09-14 06:30:06,497 INFO  [com.optinuity.c2o.ddlgenerator.InsertAndValidateDBUniqueness] [read Pool -- 97] The database type:itpamqueues is unique to the orchestrator
202X-09-14 06:30:06,497 INFO  [com.optinuity.c2o.ddlgenerator.C2ODDLGenerator] [read Pool -- 97] Database adjustment done****
202X-09-14 06:30:06,497 INFO  [com.optinuity.c2o.ddlgenerator.C2ODDLGenerator] [read Pool -- 97] Closing DB connection to: itpamqueues
202X-09-14 06:30:06,497 INFO  [com.optinuity.c2o.server.ServerController] [read Pool -- 97] Time taken for DBAdjustment : PAMJMSQueuesDB31 ms.
202X-09-14 06:30:06,513 INFO  [com.optinuity.c2o.server.ServerController] [read Pool -- 97] DDL update completed
202X-09-14 06:30:11,904 ERROR [org.jgroups.protocols.JDBC_PING] [     Thread-186] JGRP000115: Could not open connection to database
java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/RuntimeLibDS

The main key here is the "DDL update completed" entry which suggests that DB connections are being established, and there is a specific problem with one of the databases, which in this case is the entry defined in the oasisconfig.properties file, located in C:\Program Files\CA\PAM\wildfly\standalone\.config

oasis.runtime.database.dbname=PAM_RUN

Resolution

created new PAM_RUN database "PAM_RUN_NEW" in SQL Server, and modified the oasisconfig.properties to switch to the new PAM Run DB "PAM_RUN_NEW", ie:

oasis.runtime.database.dbname=PAM_RUN_NEW

The PAM access is restored

Additional Information

The following are all of the DB definitions in the oasisconfig.properties file:

oasis.database.lib.dbname=PAM_XXXX
oasis.database.queues.dbname=PAM_XXXX
oasis.reporting.database.dbname=PAM_XXXX
oasis.runtime.database.dbname=PAM_XXXX

XXXX is an arbitrary name that may be attached to each of the different PAM databases. While it is possible for there to be one database used for all of the above values, for the runtime, it is safe to define a separate blank database and assign it as the runtime database.

The above change will disrupt and cancel any in-flight processes that are in place but will enable access to PAM itself. Existing definitions will also be intact.

Further examination by Support would be necessary to determine for the specific corruption on the "PAM_RUN" database that caused the issue