Identity Portal is not running

Products

• CA Identity Governance

Issue/Introduction

Identity Portal is down. Recycled servers and nothing came back up. Portal is not running. 

 

Cause

Identity Manager server log shows "connection refused to ca-prov-srv-01:20389" during startup.

vAPP status ("s") output indicates that the clock time offsets for DSA are off by as much as 278 seconds:

                                   [WARN] clock offset of 192.0.0.1 is 278 seconds - CA Directory replication might fail

                                   [WARN] clock offset of 192.0.0.2 is 81 seconds - CA Directory replication might fail

                                   [WARN] clock offset of 192.0.0.3 is 293 seconds - CA Directory replication might fail

On the Directory Server for Prov-srv-01, we see in the DSA warning logs that multiwrite has failed and peers have been marked as down.

] 20220111.123006.820 WARN : Attempting to send update to peer 'ca-prov-srv-03-impd-co'
[2] 20220111.123006.838 WARN : MW-DISP: Sending update to 'ca-prov-srv-02-impd-co'
[7] 20220111.123006.838 WARN : MW-DISP: Sending update to 'ca-prov-srv-03-impd-co'
[2] 20220111.123357.213 WARN : comms_recv: Connection refused
[2] 20220111.123357.213 WARN : Remote DSA 'ca-prov-srv-02-impd-co' aborted
[2] 20220111.123357.213 WARN : Marking DSA 'ca-prov-srv-02-impd-co' as down
[0] 20220111.123458.885 WARN : MW-DISP not in sync for 'ca-prov-srv-02-impd-co'
[0] 20220111.123458.885 WARN : Attempting to send update to peer 'ca-prov-srv-02-impd-co'
[5] 20220111.123458.885 WARN : comms_recv: Connection refused
[0] 20220111.123559.033 WARN : MW-DISP not in sync for 'ca-prov-srv-02-impd-co'
[0] 20220111.123559.033 WARN : Attempting to send update to peer 'ca-prov-srv-02-impd-co'
[3] 20220111.123559.034 WARN : comms_recv: Connection refused
[4] 20220111.123644.751 WARN : MW-DISP: Sending update to 'ca-prov-srv-02-impd-co'
[8] 20220111.123644.785 WARN : Marking DSA 'ca-prov-srv-02-impd-co' as available again
[6] 20220111.123816.237 WARN : comms_recv: Connection refused
[6] 20220111.123816.237 WARN : Remote DSA 'ca-prov-srv-03-impd-co' aborted
[6] 20220111.123816.237 WARN : Marking DSA 'ca-prov-srv-03-impd-co' as down
[0] 20220111.123917.184 WARN : MW-DISP not in sync for 'ca-prov-srv-03-impd-co'
[0] 20220111.123917.184 WARN : Attempting to send update to peer 'ca-prov-srv-03-impd-co'
[1] 20220111.123917.184 WARN : comms_recv: Connection refused
[0] 20220111.124018.950 WARN : MW-DISP not in sync for 'ca-prov-srv-03-impd-co'
[0] 20220111.124018.950 WARN : Attempting to send update to peer 'ca-prov-srv-03-impd-co'
[7] 20220111.124043.426 WARN : MW-DISP: Sending update to 'ca-prov-srv-03-impd-co'
[1] 20220111.124043.458 WARN : Marking DSA 'ca-prov-srv-03-impd-co' as available again
[0] 20220111.124050.905 WARN : Disabling cache prior to exit
[0] 20220111.124348.126 WARN : 'clear schema;' has been disabled as it is not required
[0] 20220111.124348.526 WARN : 'min-auth'/'authentication' is depricated. Minimum setting in auth-levels is considered as min-auth.
[0] 20220111.124348.528 WARN : max-local-ops has no effect
[0] 20220111.124348.532 WARN : Loading cache
[0] 20220111.124348.555 WARN : Datastore was created at: 20210626204123Z
[0] 20220111.124348.555 WARN : Datastore was created for: ca-prov-srv-impd-co
[0] 20220111.124349.078 WARN : Cache loaded, 5801 entries
[0] 20220111.124352.076 WARN : Memory used by cache: 14846433 + 64118486
[0] 20220111.124352.077 WARN : Found new MW DSA: ca-prov-srv-02-impd-co
[0] 20220111.124352.077 WARN : Attempting to get update from peer 'ca-prov-srv-02-impd-co'
[0] 20220111.124352.078 WARN : Found new MW DSA: ca-prov-srv-03-impd-co
[7] 20220111.124352.164 WARN : Attempting to get update from peer 'ca-prov-srv-03-impd-co'
[1] 20220111.124352.252 WARN : Attempting to send update to peer 'ca-prov-srv-02-impd-co'
[1] 20220111.124352.252 WARN : Attempting to send update to peer 'ca-prov-srv-03-impd-co'
[5] 20220111.124352.272 WARN : MW-DISP: Sending update to 'ca-prov-srv-02-impd-co'
[5] 20220111.124352.272 WARN : MW-DISP: Sending update to 'ca-prov-srv-03-impd-co'
[8] 20220111.125428.900 WARN : MW-DISP: Sending update to 'ca-prov-srv-03-impd-co'
[0] 20220111.132805.852 WARN : Disabling cache prior to exit
[0] 20220111.133105.860 WARN : 'clear schema;' has been disabled as it is not required
[0] 20220111.133106.240 WARN : 'min-auth'/'authentication' is depricated. Minimum setting in auth-levels is considered as min-auth.
[0] 20220111.133106.242 WARN : max-local-ops has no effect
[0] 20220111.133106.257 WARN : Loading cache

 

What has happend above is that because the servers were not in close time synchronization, the DSAs could not stay in sync with multwrite replication and DSAs running Prov-srv became inaccessible and IM could not connect with the Prov server. With Prov not available, the Identity Manager environment does not start, and Identity Portal UI will also not come up.

Resolution

Set the times on all servers to be closely synchronized with one another.

Also, add the NTP servers to the config on Vapp to ensure ongoing synchronization.

Restart the Prov DSAs.

Restart IM, IP, etc.