How to clear Provisioning Server notification queue (inbound synchronization)

Products

• CA Identity Suite
• CA Identity Manager

Issue/Introduction

Description:

In some cases, a notification sent from Provisioning Server to Identity Manager cannot be processed by Identity Manager, due to various reasons. While this notification is in place, it is blocking the queue, preventing from further notifications (updates) from Provisioning to reach and be processed by Identity Manager.

 

The procedure below describes the procedure of identifying the relevant notification and removing it, so the queue is again functional and able to process subsequent pending notifications.

 

Environment

All versions of Identity Suite (Identity Manager)

Resolution

Identifying Use case: 

Q: First how do we identify whether the notification queue is blocked?

A: By reviewing the latest etanotifyxxxxxxxx-xxxx.log under (..\CA\Identity Manager\Provisioning Server\logs) search for the "Notifications Processed" information

START: Notify Batch Processing

 

Sending Notification: eTNotifyOpID=1da866bf-3b1f-4a0d-xxxx-4612099ceba1

 

Event: Delete_Provisioning_Object (eTDYNDirectoryName=<directory name>)

 

SeqNo: 0000000514

 

Try sending payload to 

https://<im.domain.com>:8443/idm/ETACALLBACK/?env=<env name>

 

ERROR: No message provided

 

Error in notification processing: Reason: Operation failed. ERROR: IMS was

not able to consume the notification successfully.

 

Originated from: .\EtaNotifyTools.cpp [1102].

 

DONE: Notifications Processed: 0/100+ [FAILED]

The line above states that this notification processing has failed and that there are > 100 notifications pending in the queue

Removing blocking notification:

 

Next step would be to connect to the notification DSA (<HOSTNAME>-impd-notify) using an LDAP browser and remove the problematic notification:

 

Note:  Please make sure to have a backup of the DSA in place before making any of the following changes!

 

1.  Connect to the notification DSA - default port 20404 on the Provisioning Server hostname using the user DN eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=im,dc=etadb together with the password used during installation.

(You could also lower security temporarily by setting min-auth=none; via DSA Management Console for the DSA)

 

2.  From the etanotifyxxxxxxxx-xxxx.log, identify the eTNotifyOpID of the problematic notification. From the example given earlier, we can see ours is

eTNotifyOpID=1da866bf-3b1f-4a0d-xxxx-4612099ceba1

 

3.  Using your LDAP browser session, search for it and once found delete it.

Next you should observe the rest of the pending notifications being processed (by reviewing the updated etanotifyxxxxxxxx-xxxx.log). Should you encounter further "faulty" notifications blocking the queue, steps 1-3 mentioned earlier are to be followed for each one.

 

Additional Information

If the number of pending notifications is too high and they are of no importance, there is a way of clearing the database of all notifications at once.

Please be aware you will delete all inbound notification so this operation has a potential of data loss if any notification has importance.

This can be done by following these steps:

1. Stop IM
2. Stop the Provisioning Server
3. Stop the notify DSA (<hostname>-impd-notify)
4. Keep a copy of the .db file (<hostname>-impd-notify) in an external location - for backup purposes
5. Execute dxemptydb <hostname>-impd-notify - this re-creates an empty DB
6. Start the notify DSA
7. Start the Provisioning Server
8. Start IM