New Governance servers in vApp Group will not update datasource and continue to point to wrong database: ORA-16000: database or pluggable database open for read-only access

Products

• CA Identity Manager
• CA Identity Governance
• CA Identity Portal
• CA Identity Suite

Issue/Introduction

Problem:

18:16:43,150 INFO  [stdout] (Thread-26 (HornetQ-client-global-threads-1291014334)) Caused by: Error : 16000, Position : 0, Sql = UPDATE WP_JOB_MONITOR SET ASSIGNMENT_FLAG=0, ASSIGNEE=NULL, MONITOR_FIFO=NULL WHERE (ASSIGNMENT_FLAG=1) AND MONITOR_FIFO NOT IN (SELECT MONITOR_FIFO FROM WP_MONITOR), OriginalSql = UPDATE WP_JOB_MONITOR SET ASSIGNMENT_FLAG=0, ASSIGNEE=NULL, MONITOR_FIFO=NULL WHERE (ASSIGNMENT_FLAG=1) AND MONITOR_FIFO NOT IN (SELECT MONITOR_FIFO FROM WP_MONITOR), Error Msg = ORA-16000: database or pluggable database open for read-only access

When we dig into the configuration files of the Governance vApps, they don’t recognize the database configuration change for the vApp group/cluster.

Summary:

This should not be a problem if the Governance vApps would reconfigure and take the new database server names or IP address. However, they Governance servers don’t reflect the change and continue to point to the Prod database which is read-only, so the error message is correct.

Question:

How do we get the new Governance servers in the vApp Group to reflect the datasource changes and pretend like they are part of the cluster? Initially when we added them, they configured and deployed BUT they did not show as though they were part of the vApp group/cluster. When you logged in to SSH, they told you to go to a web page to perform initial configuration. We couldn’t figure out that so we deleted them from the group and fully redeployed. On this second deployment, they think they are part of the group BUT they won’t respect changes to the External Database DB sources.

Situation:

We are testing DR failover. We are switching from an Oracle cluster in Prod to an Oracle cluster in DR. Under normal circumstances, the Oracle Cluster is running read/write in Prod. On a DR situation/test, the Oracle Cluster in DR (normally read-only) is switched to Primary and the Prod cluster is demoted to read-only.

Resolution

We found SSH key issue with one of the added IG nodes. Vapp main log shows the following when database is reconfigured.

2020 Sep 10 19:46:29 idmdprd01.abc.net GLOBAL RSA host key for ###.##.##.## has changed and you have requested strict checking.
2020 Sep 10 19:46:29 idmdprd01.abc.net GLOBAL Host key verification failed.

So we did the following steps to resolve the issue

1. Remove <IP Address> IG node from the cluster and redeploy
2. Run reset_vapp_to_factory_defaults and re-add <IP Address> IG node into the cluster and redeploy