Task Persistence Database and In Progress Tasks: Use, Performance and Troubleshooting

Products

• CA Identity Manager
• CA Identity Suite

Issue/Introduction

Ensuring the health of CA Identity Manager databases in large environments is essential to avoid performance issues experienced from rapid data growth\ sizing issues. The product documentation is a great source of information, however, this knowledge document aims to pull together a number of resources to offer guidance on maintaining the task persistence data and managing in-progress tasks.

Environment

CA Identity Manager 12.6.x
CA Identity Manager 14.x
CA Identity Suite 14.x

Resolution

Task Persistence Database: Use, Performance, and Troubleshooting:

Intended Use

• Internal task and event state tracking until task completion.
• Stored in a database with the intention of immediately deleting the Task Persistence data when the task completes.
• Writes huge number of rows and performs high r/w activity to DB by design, therefore should be cleaned up promptly and frequently.
• User viewable via View Submitted Tasks for verifying task completion and for troubleshooting assistance.
• View Submitted Tasks (VST) provides insight into the Task Persistence (TP) data as a simple way to monitor recent task completion/failure.
• See Task Persistence Intended Use in Task Persistence Database and View Submitted Tasks- Intended Purpose

Not intended for use as:

• Audit data
• Reporting data
• Compliance data
• For audit, reporting and compliance use IDMs reporting and audit functionality, do not use task persistence data
• Not intended to be accessed by custom code for the following reasons:

§  Can interfere with the performance
§  Can cause DB lock conditions
§  Avoid writing custom code or PX policies that access TP database

• See Task Persistence Intended Use in Task Persistence Database and View Submitted Tasks- Intended Purpose

Performance Implications

• Every task and event of each task has states that change from one processing step to the next, each of which is tracked in TP. This requires many DB connections and high r/w activity for each task.
• The larger the TP database grows, the longer the queries take, which can significantly impact system performance.
• A large TP database with heavy activity can become a performance bottleneck in which tasks appear to be processing slowly or seem to remain ‘in-progress.’

Database Space Usage

• Database space requirements vary depending on task activity and task complexity
• See KB 32151 - Task Persistence Database Disk Space Sizing

You can minimize the amount of data in TP tables using the following methodologies:

• Run Cleanup Submitted Tasks

§  Use for ongoing cleanup
§  Not effective for massive cleanup
§  Run daily or weekly keeping only the most recent few days of data
§  Removes only Completed tasks—no risk of removing in-progress tasks

•  Keep only the most recent tasks
• See documentation for configuring Cleanup Submitted Tasks. (Cleanup Submitted Tasks)
• For mass cleanup, run stored procedure, “garbageCollectTaskPersistence”

§  Stored procedure removes only completed tasks, not in-progress tasks.
§  Stored proc is installed into Object Store
§  Avoid running during busy hours
§  Can take a long time—run small batches
§  See the Readme and SQL Script for running the stored procedure in
..\CA\Identity Manager\IAM Suite\Identity Manager\tools\samples\TaskPersistence\SQLServer
§  See KB 43199, Cleanup Submitted Tasks Not Working; Database stored procedure for cleaning up task persistence is available.

Stuck In-progress Tasks

• Can be slow and not progressing due to bottleneck
• o   Or permanently stuck due to task state inconsistency (system failure, for example)
• o   Tasks stuck in progress can be resubmitted with the Post-Events feature (available in IDM12.6.8 or later)

§  The Post-Events feature allows the user to resubmit events that get stuck in-progress. This feature is accessible via the Post Events button on the Identity Manager Environment Properties page.
§  See Post Events to the Identity Manager Event Service in Post Events to the Identity Manager Event Service

• Tasks sometimes become stuck due to inconsistent state and corruption cannot be restarted; cannot be cleared via View Submitted Tasks

§  It’s OK to just leave them as is.

Overview of Diagnosing Stuck In-Progress Tasks
   for more specific information on In-Progress Tasks please see KB 233003 IN-PROGRESS task issues - A Client’s Guide to Understanding and Resolving

o   Usually due to a processing bottleneck

§  Bulk feed processing?
§  Explore Correlate running?
§  Check VST, look for patterns of stuck tasks
§  What is the last event (password sync to prov, for ex.)
§  Check for Task Persistence DB slowness
§  User directory or Object store DB connection issues
§  Check IDM application server logs for problems
§  Provisioning server issues

·         Check Prov logs (etatrans, etanotify, JCS)
·         Inaccessible endpoints
·         Connector server not running
·         CA directory errors
·         Outbound sync (to IDM) failures to ETACallback servlet
·         Notifications not processing, or not keeping up with load

§  Check Java Messaging Service (JMS) processing for problems