SDM 14.1.x - Animator crashes repeatedly and does not restart
Article ID: 136354
Updated On:
Products
Issue/Introduction
In Service Desk Manager 14.1.x, the Animator process (animator_nxd) handles all events and scheduled tasks, including auto-closure, scheduled notifications, and service level agreements, among other tasks. The Animator stores all of its' tasks in a database table called 'anima', and queries that table to determine when events should be fired.
Under certain specific conditions, timestamp data can be inserted into the 'anima' table that cannot be properly parsed by the Animator process, causing it to crash. Normally, the process would restart up to 10 times (by default) when it crashes, but under these conditions, the process will start and immediately crash again until it reaches that maximum retry count, after which point the Animator will stay in a Not Running state. While the Animator is Not Running, scheduled tasks and time-based events will no longer function. During this behavior, you may see repeated messages about singleton processes or invalid process IDs, especially on servers other than the one where the Animator normally runs.
Environment
This behavior is seen in Service Desk Manager versions 14.1.x and earlier.
Service Desk Manager versions 17.x and newer are unaffected.
Cause
If the 'anima' table includes a timestamp in the 'a_time' or 'a_org' columns that is negative, the Animator process will fail to start.
The 'anima' table stores the time at which an event should be triggered in standard epoch (Unix) time, which stores times as the number of seconds since 1st January 1970. If a date earlier than 1st January 1970 is needed, it is stored as a negative integer in the database. Due to the size of the signed integer, the latest possible timestamp that can be stored is for 03:14:07 GMT on 19th January 2038; attempting to store a timestamp later than that will cause an overflow, resulting in a negative integer that will commonly be parsed as 13th December 1901 instead. This is broadly known as the "Year 2038 Problem", and is a known issue in computing. Regardless of whether a timestamp is intentionally backdated prior to 1970 or is inadvertently negative due to being further out than 2038, the Animator process in SDM 14.1.x is not able to properly handle timestamps with a negative value, and will crash if they are present in the 'anima.a_time' or 'anima.a_org' columns.
Resolution
As long as the negative integer remains in the 'anima.a_time' or 'anima.a_org' columns, the Animator will be unable to start; it is necessary to either remove or alter the affected rows in the database to proceed. Before proceeding, it is recommended to take backups of both your database and your SDM server for rollback purposes. It is strongly recommended to follow the first series of steps below, as they are the supported way to insert and update database data; the second set of steps is provided only for advanced users who prefer an unsupported approach.
1. Using pdm_extract and pdm_load
1.1. The pdm_extract command can extract the contents of the 'anima' table, allowing you to locate the rows with negative timestamps. The -f flag allows you to specify a SQL select statement for your extract. Run the below command to extract the data (note that the "Animator" portion of this command is case-sensitive):
pdm_extract -f "SELECT * FROM Animator WHERE (a_time<0 OR a_org<0)" > Animator.txt
Note: If this command returns an error, you may not have negative timestamps in your 'anima' table and are experiencing a different issue. Please contact Support if this occurs.
1.2. After the extract is complete, review Animator.txt and confirm that the 'a_time' and 'a_org' columns are indeed negative, and that no extraneous rows are included.
1.3. For each row, edit the value of the 'a_time' and 'a_org' columns. You can either set both of these values to 0 (which will cause the event to have a firetime of 1st January 1970, and may look odd in reports), remove the leading "-" (which will also likely look odd in reports), or set them to the current date and time (online tools exist to report on the current epoch time). Save your changes to Animator.txt.
1.4. The pdm_load command can insert data into the database with a given input file. The -u flag will update all matching rows without inserting new ones, while the -f flag specifies your input file. Use the command below to overwrite the existing rows in the database with your modified Animator.txt file:
pdm_load -u -f Animator.txt
1.5. Restart the Animator process by running the pdm_d_refresh command. You can check the status of the process by running the pdm_status command and looking at the "Animation" row near the top.
2. Directly edit the 'anima' table
Warning: This method is completely unsupported, as directly modifying the database can have unexpected effects (up to and including loss of data or complete system failure). Take backups of both your SDM server and database before making any direct database changes.
2.1. Using your database management tool of choice, open up your 'mdb' database.
2.2. Run the following query to isolate any rows in the 'anima' table with negative timestamps:
SELECT * FROM anima WHERE (a_time<0 OR a_org<0)
2.3 Update the affected rows to correct the negative timestamp. As noted above, you can either set this to 0 (1st January 1970, looks odd in reports), remove the leading "-" (may have an unexpected date and could look odd in reports), or set it to the current date and time (using an online tool to obtain the current epoch timestamp). Replace the [your timestamp here] section in the below query with your chosen timestamp.
UPDATE anima SET a_time = [your timestamp here], a_org = [your timestamp here] WHERE (a_time<0 OR a_org<0)
2.4. Re-run the SELECT statement from step 2.2 to confirm that no rows are returned.
2.5. Restart the Animator process by running the pdm_d_refresh command. You can check the status of the process by running the pdm_status command and looking at the "Animation" row near the top.
After following the steps above, your Animator process should be functioning as normal. Depending on how long the process has been offline, it may have a backlog of events to process; during this time, it is normal and expected to see messages in your stdlog files indicating that the "firetime of atev" was missed, or that the Animator has a number of backlogged or missed events. The Animator will process all of its' missed events and should catch up eventually, especially during periods of low system usage. If you have a very large number of missed events, you may experience some system performance degradation while the Animator works through its' backlog.
Additional Information
For more information on the Year 2038 Problem:
https://en.wikipedia.org/wiki/Year_2038_problem
For more information on the 'anima' table:
For more information on the pdm_extract and pdm_load commands:
https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=804055
Feedback
