Lookback API shows discrepencies when compared to revision history

Products

• Rally SaaS

Issue/Introduction

When examining some lookback API snapshots a couple of different types of discrepancies can appear when comparing against revision history.

• Missing snapshots
• Snapshots out of order

Environment

Release : SAAS

Component : ACAPI

Cause

Some of these discrepancies are caused by a time synchronization issue between the application servers that was discovered in April 2021.  This time synchronization issue was resolved and we now synchronize time to 5 decimal places.

The issue could appear when there was a time difference of several seconds and multiple change requests would appear across multiple application servers.  Each application server would mark their change with their timestamp.  The revision times would be out of order and fed to the lookback API in this order.

When lookback API gets these changes, one of two things may happen depending on the revision number.

1. If the first snapshot and second snapshots are out of order, we would drop the second snapshot because there is no first "root" snapshot.  This would appear as a missing snapshot
2. If the second or higher snapshots have times that are out of order, then the lookback API will present these as being out of order

 

Resolution

There is no resolution to for snapshots that are currently out of order.

Since April 2021 it is less likely to encounter this scenario as we are now synchronizing time with a much higher resolution, however, it is still technically possible to encounter this situation.

If an integration is performing these changes rapidly back to back on a single work item, then it is highly recommended to have the integration utilize cookies and capture the JSESSIONID attribute and reuse this.  This will keep the integration pinned to a single application server and reduce the likelihood of multiple requests being serviced by multiple application servers.