Symptoms:
After upgrading from 9.1 to 9.7 "CEM|TESS Agent" metrics have stopped appearing in the Investigator and the EM running Tim Collection Service shows a log error trying to start the BtStatsPollTask :
[ERROR] [TimPollThreadPool.Thread6] [Manager.com.timestock.tess.util.DbUtils] Select failed for com.timestock.tess.data.objects.Setting
org.hibernate.NonUniqueResultException: query did not return a unique result: 2
at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:813)
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:804)
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4551)
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4504)
at com.timestock.tess.util.DbUtils.getSettingValue(DbUtils.java:9049)
at com.timestock.tess.services.tim.BtStatsConfigState.getSetting(CreateBtStatsConfig.java:163)
at com.timestock.tess.services.tim.BtStatsConfigState.createBtStatsConfigDocument(CreateBtStatsConfig.java:129)
at com.timestock.tess.services.tim.CreateBtStatsConfig.createBtStatsConfigDocument(CreateBtStatsConfig.java:76)
at com.timestock.tess.services.tim.TimIo.sendBtStatsConfig(TimIo.java:2460)
at com.timestock.tess.services.tim.TimIo.enableBtStats(TimIo.java:2291)
at com.timestock.tess.services.tim.BtStatsPollTask.onStart(BtStatsPollTask.java:48)
at com.timestock.tess.services.tim.TimPollTask.run(TimPollTask.java:165)
...
Cause:
9.7 has the new "Enhanced RTTM by User Group" feature (RTTM=BtStats).
Some changes to the APM DB involve adding 4 rows to the TS_SETTINGS table for these TS_KEY values:
btstats.enabled, btstats.urlpathenabled , btstats.usergroupenabled, btstats.maxstatspertim.
This problem was caused by those 4 rows being duplicated, possibly due to a second upgrade step on the APM DB.
Resolution:
1. Run this SELECT SQL:
SELECT TS_ID, TS_KEY FROM TS_SETTINGS WHERE TS_KEY LIKE 'btstats%';
The results should verify that more than set of the btstats rows exists.
2. Stop the EM running the Tim Collection Service
3. For the 4 duplicate TS_KEY btstats rows delete each one by specifying its corresponding TS_ID value in this DELETE SQL:
DELETE FROM TS_SETTINGS WHERE TS_ID IN(ts_id1, ts_id2, ts_id3, ts_id4)
If using an Oracle APM DB also COMMIT the change
4. Restart the EM running the Tim Collection Service.
5. Check the EM log for any errors relating to the BtStatsPollTask and the Investigator for expected "CEM|TESS Agent" metrics.