How are the CEM stats percentile values that are displayed in CEM reports/graphs calculated.


Article ID: 35407


Updated On:


APP PERF MANAGEMENT CA Application Performance Management Agent (APM / Wily / Introscope) CUSTOMER EXPERIENCE MANAGER INTROSCOPE



How are the CEM stats percentile values displayed in CEM reports and graphs calculated.



The TIM performs calculations on the individual transactions and those statistics are sent to the EM in an hourly .stats file. (Data for individual defects are stored and processed in a different manner)

For performance reasons the TIM does not send all the individual data points to the EM/TESS to be stored in the APM Database.

Rather the .stats file the TIM distributes the 3 types of data (time, throughputsize) into 24 separate "buckets" using a V-optimal histogram approach which is described here: V-optimal histograms

Each bucket value represents the maximum value recorded for a group of data points ("count") and that bucket value increases with bucket number.

When percentile data has to be displayed on the CEM UI:

  • The MOM runs the final percentile algorithm.
  • The algorithm calculates which bucket contain the percentile data point and uses its data value as the final percentile value.
  • The final percentile value is not stored in the APM Database and is always created on the fly.


Detailed Additional Information:

The .stats files are created into directory …/tim/data/out/stats on the TIM file system and are deleted as the EM processes the files.

An example .stats file has the following information:

Stats record 1 at offset 0 (0x0)

AppDefId = 700000000000000000

TranSetId = 700000000000000000

TranUnitId = 0

TranCompId = 0

TranSetIncarnationId = 3112934926792304

Version = 8

IntervalStartTime = Wed Jan 4 13:00:00 2012

IntervalWireTime = 3600 seconds

DataType = 1

TotalTxCount = 37258

BadTxCount = 29935

OpportunityCount = 223548

DefectCount = 29935

Time histogram at offset 72 (0x48)

MaxValue = 13234

Count = 37258

SumOfSquares = 0

SumOfValues = 15260537

Throughput histogram at offset 392 (0x188)

MaxValue = 680100

Count = 37258

SumOfSquares = 0

SumOfValues = 3169500744

Size histogram at offset 712 (0x2C8)

MaxValue = 525329

Count = 37258

SumOfSquares = 0

SumOfValues = 348481060

SessionId =

LoginId =

UserGroup = UserGroupTim-


Stats record 2 at offset 1080 (0x438)

AppDefId = 700000000000000000

TranSetId = 700000000000000004

TranUnitId = 0

The EM moves the stats file data into the APM DB tables.

Base tables are ts_st_ts_us_int_<current date> for transet data (Business Transactions) and ts_st_tu_us_int_<current date> for tranunit data (Transactions). NOTE: Due to potential hgh data volumes by default only Business Transactions statistics are enabled when a definition is created in CEM.

From there, the data is migrated into (hourly tables) and on a daily basis, the dly, wly, and mly tables.

Each statistics record stored contains 24 buckets of data for three histograms:

  • one for time (columns: ts_tth_bn, ts_tth_cn),
  • one for throughput (columns: ts_tph_bn, ts_tph_cn)
  • one for size (columns: ts_tsh_bn, ts_tsh_cn)

where n=0,...,23

Each ts_*_bn value represents the maximum value recorded for ts_*_cn data points (the "count") contained in that bucket and the ts_*_bn value increases with the value of e.g. 

ts_tth_b(0:23)= {264,266,267,271,272,274,276,277,284,286,287,292,294,301,305,309,319,320,322,325,329,341,350,363};


The final percentile algorithm on the TESS performs these steps:

  • Total the counts from all buckets to get the total number of data points and multiply it by the desired percentile to obtain the percentile data point.
  • From a running total of all counts finds which bucket contains that percentile data point & use its value as the final percentile value.


Additional Information:

It was recently uncovered that an extra calculation on the final percentile value to weight adjust it with the corresponding value in the adjacent higher bucket could cause an unexpected skew to that value if that higher bucket has an unusually high maximum value (due to some very long running transactions). So that has recently been fixed and the change will be available in a future release of APM.


Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main