This week, we discuss how to configure Identity Manager (IM) for submitted task cleanup (STC) in order to manage database growth. During health checks for existing IM deployments, we often find that they do not have STC, a critical maintenance task, enabled. When active, it is responsible for making sure that the task persistence database does not grow too large; if not active, it could result in an exhausted temporary table space.
There are a number of factors to consider when creating cleanup jobs. First, breaking the operations into two separate jobs is not required, but using two different jobs allows for longer job execution times (when Identity Manager is not heavily utilized) and cause less contention for database resources.
Next, before scheduling any task cleanup activities, make sure that the required information to properly schedule the task is available. We believe that you will need to consider the following items before scheduling any tasks:
- Database maintenance schedule
- Server / database backup schedule
- Number of Identity Manager environments
Also, the task cleanup event should not be scheduled at a time that database maintenance is occurring. Additionally, it is important to schedule this activity outside of the backup window for the servers involved. Since the task cleanup must be run for the Identity Manager environment, we believe that cleanup tasks should be staggered to ensure there is no contention.
The considerations listed earlier impact the schedule of the STC, but the following items should be considered to help determine the scope and method of the cleanup:
- Auditing requirements for identity related activities
- Archival requirements
- Database constraints
Many organizations require the audit data to be retained for a specific number of years before being deleted. If your archive database has enough space to accommodate the retention period, the data can be archived before deletion without encountering storage issues. If not, once the tasks are removed from the runtime database, they will no longer appear when searching the submitted tasks in Identity Manager. Understanding your organization’s or industry’s auditing requirements will help determine how the cleanup tasks are configured regarding minimum age and task archiving.
It is also important to speak with the database administrator to understand what capacity limitations exist. For example, you may want to process fifteen thousand tasks every day at 11 PM, but another application could be doing batch processing at that time and consuming a large amount of database server/cluster resources.
Configuring STC in Identity Manager
To configure STC, select the entry under the system menu in Identity Manager.
Select the Schedule new job option. The next two sections of this post correspond to the two configuration tabs within the STC configuration menu.
Recurrence
In this section, we will configure the frequency and execution time for the STC. The following steps outline scheduling for weekday and weekend jobs.
- Enter the appropriate job name for this activity
- Choose the time zone for your servers
- Select the type of schedule that is important
Note: make sure that you select days that do not conflict with other database workloads or maintenance windows.
- For a weekday schedule, select Weekly Schedule plus the following:
- Select weekdays
- For a weekend schedule, select Weekly Schedule plus the following:
- Select Saturday and Sunday
- Enter the Execution time
Once again, make sure that the time you select does not conflict with other network events or database activities.
Now that we have completed STC scheduling, we move on to configuring the cleanup activity itself.
Cleanup Submitted Tasks
In order to help you determine the appropriate values based on the considerations we discussed earlier, the following list covers the various parameters and how to set them.
- Minimum Age – The value selected for this parameter should be aligned with your organization’s requirements. The minimum age limit will force tasks older than that value to be deleted. It’s important to understand that not all tasks older than the minimum age will be deleted; the number of tasks deleted depends on the execution parameters.
- Cleanup Via Stored Procedure – This option is enabled by default and it is the fastest way to clean up submitted tasks. Using this option will not provide progression during its execution.
- Archive – If you your organization’s audit requirements stipulate the preservation of identity tasks for a period that is longer than the minimum age (defined in number 1 above), you will need to enable the archive option. We only recommend using this option if you have installed Identity Manager with the separate database option. This option provides a dedicated archive DB for task storage.
- Audit Timeout – The value of the audit timeout determines how long unsubmitted tasks will remain active. Only tasks that are completed or have failed can be deleted.
- Time Limit – This option is only available if the “Cleanup via Stored Procedure” option is not active and it is used to limit how long the cleanup task executes. If you enable this option, make sure that the time limit entered will not conflict with other activities such as server and database backups.
- Task Limit – This controls how many tasks will be processed during the cleanup. We need to keep in mind that without a time limit, it will be hard to determine when the task cleanup will complete. The speed of task cleanup depends on several factors; selecting the correct value for the task limit may take several iterations in order to ensure that it completes before the maintenance window ends. Additionally, consult with your organization’s DBA before setting this value.
- Delete per Transaction – This value limits how many tasks are deleted per transaction. We typically set this value to 10% of the total task limit.
- Cancel in Progress Tasks Only – This option will cancel in progress tasks so that they will become eligible for cleanup in the future. We generally do not recommend using this option because there could be tasks that are being processed and this option would cancel these tasks. There are, however, instances where this option should be used. Sometimes tasks will become stuck in a pending state and they would need to be cancelled.
- Cancel Workflow Tasks Only – This option will cancel workflow tasks that are still active, so that they would become eligible for cleanup in the future. We generally do not recommend using this option because workflow tasks can span multiple days and this option could cancel those long running workflow tasks. As mentioned above for in progress tasks, workflow tasks can become stuck in a pending state and they should be cancelled.
As always, we hope that you have found this information useful. If you need IAM assistance, reach out to SIS today and we would be happy to assist you. And please subscribe to our newsletter to be notified about the posting of future articles and other SIS news.
