The Postgres database has run out of disk space due to faulty defects, getting "No space left on device" in the MoM IntroscopeEnterpriseManager.log.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Some defects have recently grown exponentially and the Postgres database /data directory contains files much larger than expected.

 

Environment

Application Performance Management 10.7

Resolution

Any misconfiguration of a Business Service transaction can lead to this problem.

For example, "Large Size" in Transaction Specification under CEM UI > Administration > Business Services that's set to >= 10 kb can create a large number of defects and cause the "no space left on device" problem.

• Determine if there are a large number of defects by looking at the incidents/defects in CEM UI > CEM > Incident Management.
• Disable the associated Defect Specifications in CEM UI > Administration > Business Services to avoid further increase in disk space usage first.
• To reduce disk space that's already taken up by the faulty defects, run these SQL statements against the APMDB to remove them:

* Important Note: be sure to backup the APMDB and stop all EMs (MoM and all Collectors) before execution.

DELETE FROM ts_tcp_session_data WHERE ts_defect_id IN (SELECT ts_id FROM ts_defects_);DROP TABLE ts_defects_;alter table ts_defect_meta_values_drop constraint FKE00FC8E3955976_;DROP TABLE ts_defect_meta_values_;DROP TABLE ts_tran_comp_details_; *  should be replaced by yyyymmdd, for example,-- to drop tables for 2013-04-05DELETE FROM ts_tcp_session_data WHERE ts_defect_id IN (SELECT ts_id FROM ts_defects_20130405);DROP TABLE ts_defects_20130405;alter table ts_defect_meta_values_20130405 drop constraint FKE00FC8E3955976_20130405;DROP TABLE ts_defect_meta_values_20130405;DROP TABLE ts_tran_comp_details_20130405;

Consider re-setting the "Delete Defects after" under CEM UI > Setup > Incident Settings to 2 days so the EM can drop big partitions overnight.

After a day, increase this back to 7 days (default) to retain data normally.