CA Application Performance Management Agent (APM / Wily / Introscope)
INTROSCOPE

**Question:**

How are the CEM stats percentile values displayed in CEM reports and graphs calculated.

**Answer:**

The TIM performs calculations on the individual transactions and those statistics are sent to the EM in an hourly .stats file. (Data for individual defects are stored and processed in a different manner)

For performance reasons the TIM does not send all the individual data points to the EM/TESS to be stored in the APM Database.

Rather the .stats file the TIM distributes the 3 types of data (**time**, **throughput**, **size)** into 24 separate "buckets" using a **V-optimal histogram** approach which is described here: V-optimal histograms

Each bucket value represents the maximum value recorded for a group of data points ("count") and that bucket value increases with bucket number.

When percentile data has to be displayed on the CEM UI:

- The MOM runs the final percentile algorithm.
- The algorithm calculates which bucket contain the percentile data point and uses its data value as the final percentile value.
- The final percentile value is not stored in the APM Database and is always created on the fly.

**Detailed Additional Information:**

The .stats files are created into directory *…/tim/data/out/stats* on the TIM file system and are deleted as the EM processes the files.

An example .stats file has the following information:

*Stats record 1 at offset 0 (0x0)*

*AppDefId = 700000000000000000*

*TranSetId = 700000000000000000*

*TranUnitId = 0*

*TranCompId = 0*

*TranSetIncarnationId = 3112934926792304*

*Version = 8*

*IntervalStartTime = Wed Jan 4 13:00:00 2012*

*IntervalWireTime = 3600 seconds*

*DataType = 1*

*TotalTxCount = 37258*

*BadTxCount = 29935*

*OpportunityCount = 223548*

*DefectCount = 29935*

*Time histogram at offset 72 (0x48)*

*MaxValue = 13234*

*Count = 37258*

*SumOfSquares = 0*

*SumOfValues = 15260537*

*Throughput histogram at offset 392 (0x188)*

*MaxValue = 680100*

*Count = 37258*

*SumOfSquares = 0*

*SumOfValues = 3169500744*

*Size histogram at offset 712 (0x2C8)*

*MaxValue = 525329*

*Count = 37258*

*SumOfSquares = 0*

*SumOfValues = 348481060*

*SessionId =*

*LoginId = 10.78.100.106*

*UserGroup = UserGroupTim-10.72.0.0*

* *

*Stats record 2 at offset 1080 (0x438)*

*AppDefId = 700000000000000000*

*TranSetId = 700000000000000004*

*TranUnitId = 0*

*…*

The EM moves the stats file data into the APM DB tables.

Base tables are **ts_st_ts_us_int_ <current date>** for transet data (Business Transactions) and

From there, the data is migrated into (hourly tables) and on a daily basis, the dly, wly, and mly tables.

Each statistics record stored contains 24 buckets of data for three histograms:

- one for
**time**(columns:**ts_tth_b****n**,**ts_tth_c**),*n* - one for
**throughput**(columns:**ts_tph_b***n**,***ts_tph_c**)*n* - one for
**size**(columns:**ts_tsh_bn***,***ts_tsh_c**)*n*

where * n*=0,...,23

Each **ts_*_b n** value represents the maximum value recorded for

**ts_tth_b****(0:23)****= {264,266,267,271,272,274,276,277,284,286,287,292,294,301,305,309,319,320,322,325,329,341,350,363};**

**ts_tth_c(0:23)****={1,3,5,1,2,6,3,5,1,2,4,1,1,1,1,2,1,1,1,1,2,1,1,1};**

The final percentile algorithm on the TESS performs these steps:

- Total the counts from all buckets to get the total number of data points and multiply it by the desired percentile to obtain the percentile data point.
- From a running total of all counts finds which bucket contains that percentile data point & use its value as the final percentile value.

**Additional Information:**

It was recently uncovered that an extra calculation on the final percentile value to weight adjust it with the corresponding value in the adjacent higher bucket could cause an unexpected skew to that value if that higher bucket has an unusually high maximum value (due to some very long running transactions). So that has recently been fixed and the change will be available in a future release of APM.

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main

Component:

Component:

Yes

No