Missing tasks and different environment IDs

Products

• CA Identity Manager

Issue/Introduction

In the Identity Manager environment, there were missing email tasks, password change tasks, and other unexpected user environment problems.

The environment ID was not correct, as if the environment had been deleted and recreated.

Cause

The environment OID had been changed three times in 12 hours. This was causing the missing email tasks, password change tasks, and other user environment problems. We do not know how the environment OID was getting changed. There is nothing in the IM code that can delete or create an IME.

This would typically occur if the IME was manually deleted and recreated, in which case a new OID is assigned. This could happen via the IM Management console on the Prod server, or via another IM system pointing to the Prod database.

According to entries in the tasksession12_5 table we noted the approximate times of tasks running on the different environment IDs to get a general idea of when the IME IDs changed:

EnvID  2    June 16 - 7:14am
Evn ID 5    June 16 - 8:00 am
Env ID 6    June 16 - 8:50am
Env ID 9    June 17 - 8:00am

IM/Wildfly logs do not indicate when the Env IDs were changed or show any behavior indicating an IME was updated/deleted/created, though there is not normally any logging of such activity.

There are errors indicating incorrect Env IDs but these are after reboots when IM is attempting to load object store data. We would expect to see such errors when the IM Environment ID is incorrect.

The root cause is not known. The most common scenario is either another IM installation (like a QA or Sandbox, for example) pointing to the production DB and making IME changes, or changes to the environment made directly on the Prod server via the IM Management console.

Resolution

Restore the Identity Manager database to its last known working date.

 

Once the environment is confirmed to be stable, ensure your system is secure including changing IM system manager password(s), and protecting the IM Management console via SiteMinder.