REPORT MANAGER CONTAINS STALE BUCKETS in Spectrum 10.4.x

Products

• CA Spectrum

Issue/Introduction

There are too many unprocessed files in the reporting folder of SRM, resulting in the following alarm being generated:

"REPORT MANAGER CONTAINS STALE BUCKETS"

Environment

Spectrum 10.4.x and later

Resolution

Implemented the following to identify the device and corresponding NCM events causing the issue, then delete them from the DB:

1. Get the earliest NCM bucket tables for each landscape from the bucketactivitylog table. So at the command line:
   

   cd $SPECROOT/mysql/bin
   
   

2. Enter MySQL:
   

   ./mysql -uroot -proot reporting
   
   

3. At the MySQL command prompt, run;
   
   

   select * from bucketactivitylog where bucket_table_name like "%ncmconfig%" and destroy_time is null group bylandscape_h order by creation_time asc;

   
   
4. For each table that is listed in step 3, check if that table exists in the DB as, so for each example:
   

   show tables like <ncmconfigbucket33554432_1515765462386>;
   

   
   
5. Get the first record from all of the above tables. It will give you the modelKey and modelH.
   

   select * from  <ncmconfigbucket33554432_1515765462386> limit 1 \G

    

6. Check the corresponding event in the OC as well. One of the bucket record/event causing the issue with ConfigMH is null in the event msg.
   
   
7. Delete the event from the current bucket whichever has this matching OC event with configMH is null:
   

   delete from <ncmconfigbucket33554432_1515765462386> where event_seq = ?;
   

   Replace the '?' with the value of event_seq from the output of step 5:

   select * from  <ncmconfigbucket33554432_1515765462386> limit 1 \G

   The query above should give you the complete record from the bucket table. One of the columns is event_seq. It is a unique id to delete that error record.

8. Restart OneClick (Spectrum Tomcat) to verify bucket processing is started.