ITPAM service went down and will not come back up again

Products

• CA Process Automation Base

Issue/Introduction

Our ITPAM services went down and will not come back up. I ensured the agent that ties those servers to the main service desk servers was up and running, but on the ITPAM servers themselves, the CA Process Automation Orchestrator service will start running and go down within 30 seconds.   

Environment

Release : 4.3

Cause

The database user that they were using had lost access to all ITPAM databases

Resolution

The customer reached out to their DBA group and they found that the database user that they were using had lost access to all ITPAM databases.  Once they reinstated this access everything kicked off and started working as expected.  

C2o log shows:

Caused by: org.jboss.util.NestedSQLException: Unable to get managed connection for RuntimeLibDS; - nested throwable: (javax.resource.ResourceException: Unable to get managed connection for RuntimeLibDS)
    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:95)
    at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69)
    at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:421)
    ... 88 more
Caused by: javax.resource.ResourceException: Unable to get managed connection for RuntimeLibDS
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:441)
    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
    ... 90 more
Caused by: org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Cannot open database "PAMRUN" requested by the login. The login failed.)
    at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:225)
    at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:195)
    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:633)
    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:267)
    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:679)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
    ... 94 more
Caused by: java.sql.SQLException: Cannot open database "PAMRUN" requested by the login. The login failed.