After upgrading from 9.1 to 9.7 "CEM|TESS Agent" (RTTM/BtStats) metrics have stopped appearing in the Investigator and EM running Tim Collection Service shows an error trying to start the BtStatsPollTask 'query did not return a unique result: 2'

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Symptoms:

After upgrading from 9.1 to 9.7 "CEM|TESS Agent" metrics have stopped appearing in the Investigator and the EM running Tim Collection Service shows a log error trying to start the BtStatsPollTask :

[ERROR] [TimPollThreadPool.Thread6] [Manager.com.timestock.tess.util.DbUtils] Select failed for com.timestock.tess.data.objects.Setting

org.hibernate.NonUniqueResultException: query did not return a unique result: 2

at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:813)

at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:804)

at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4551)

at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4504)

at com.timestock.tess.util.DbUtils.getSettingValue(DbUtils.java:9049)

at com.timestock.tess.services.tim.BtStatsConfigState.getSetting(CreateBtStatsConfig.java:163)

at com.timestock.tess.services.tim.BtStatsConfigState.createBtStatsConfigDocument(CreateBtStatsConfig.java:129)

at com.timestock.tess.services.tim.CreateBtStatsConfig.createBtStatsConfigDocument(CreateBtStatsConfig.java:76)

at com.timestock.tess.services.tim.TimIo.sendBtStatsConfig(TimIo.java:2460)

at com.timestock.tess.services.tim.TimIo.enableBtStats(TimIo.java:2291)

at com.timestock.tess.services.tim.BtStatsPollTask.onStart(BtStatsPollTask.java:48)

at com.timestock.tess.services.tim.TimPollTask.run(TimPollTask.java:165)

...

Cause:

9.7 has the new "Enhanced RTTM by User Group" feature (RTTM=BtStats).

Some changes to the APM DB involve adding 4 rows to the TS_SETTINGS table for these TS_KEY values:

btstats.enabled, btstats.urlpathenabled , btstats.usergroupenabled, btstats.maxstatspertim. 

This problem was caused by those 4 rows being duplicated, possibly due to a second upgrade step on the APM DB.

 

Resolution:

1. Run this SELECT SQL:

SELECT TS_ID, TS_KEY FROM TS_SETTINGS WHERE TS_KEY LIKE 'btstats%';

The results should verify that more than set of the btstats rows exists.

2. Stop the EM running the Tim Collection Service

3. For the 4 duplicate TS_KEY btstats rows delete each one  by specifying its corresponding TS_ID value in this DELETE SQL:

DELETE FROM TS_SETTINGS WHERE TS_ID IN(ts_id1, ts_id2, ts_id3, ts_id4)

If using an Oracle APM DB also COMMIT the change

4. Restart the EM running the Tim Collection Service.

5. Check the EM log for any errors relating to the BtStatsPollTask and the Investigator for expected "CEM|TESS Agent" metrics.

Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component: