Spectrum Report Manager (SRM) bucket files are not getting processed.

Products

• Spectrum

Issue/Introduction

Recently created bucket files in the reporting directory do not seem to be processing. There are no corruption errors in the tomcat log and MYSQL.out shows no errors.
Some records are being processed, but slowly, resulting in a large number of unprocessed bucket table files building up.

In busy environments, there is usually a build up of buckets, waiting on an insert to the reporting DB, but the MySQL process list shows that there is no waiting on a queue of inserts.

mysql> show full processlist; 

Looking at the bucket files in the reporting directory, all are model availability buckets. For example:

-rw-rw----. 1 nmsdist users 8854    Sep 25 09:36 modelavailbucket1092616192_1537860988424.frm 

-rw-rw----. 1 nmsdist users 5504160 Sep 25 09:56 modelavailbucket1092616192_1537860988424.MYD

The buckets are not showing in the bucket activity log when queried in SRM.

Environment

Release: Any
Component: SPCRPT

Resolution

As the build-up of bucket files increases continuously, other buckets are being processed. With no corruption errors in the tomcat log and nothing the MySQL log, the DB seems okay.
Looking further at the tomcat log we an error on one of the archive managers.

Sep 27, 2018 11:19:13.019 (SRM/EventPoller/xxxxx) (SRM_Events) - (ERROR) - Primary ArchiveManager seems to be down and connected to the Non-Primary ArchiveManager. and Event processing is not supported in this case. So, skipping this processing cycle. 

Resolving this Archive Manager problem will allow the bucket files to be processed. This can be confirmed by checking how and when the bucket files were processed in the bucketactivity log in SRM at the MySQL command line.  The example below shows a query for a bucket file table (modelavailbucket16777216_1530705239861) listed in the ~/mysql/data/reporting directory. To check for others, change it for one from the list of previously unprocessed buckets in the logs at mysqlreporting.out.

mysql> select * from bucketactivitylog where bucket_table_name like 'modelavailbucket16777216_1530705239861'; 

+--------+--------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+ 

| log_id | bucket_table_name                    | landscape_h | handler_name     | finished_inserting_events | creation_time       | destroy_time.       | 

+--------+--------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+ 

| 1253   |modelavailbucket16777216_1530705239861| 16777216    | modelavailbucket | 1                         | 2018-07-04 13:53:59 | 2018-07-04 13:54:00 |

+--------+--------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+

1 row in set (0.04 sec) 

The above shows the Destroy time (2018-07-04 13:54:00 ) which means the bucket table was processed and subsequently removed at this time. The creation time can also be used to query based on the timestamps on the bucket files we have in this directory listing. For example: 

select * from bucketactivitylog where creation_time BETWEEN '2018-09-25 09:36:00' AND '2018-09-27 10:35:00'; 

Additional Information

Logs needed when investigating bucket files.

1. Run the following on MySQL of the OC SRM server: from a shell, navigate to the $SPECROOT/MySQL/bin directory and launch the MySQL command prompt: 
   
   

   ./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> reporting  

2. Run the following command to redirect the MySQL output to a TEXT file: 

   mysql> \T queries.txt 
   
   

3. Run the following MySQL queries: 
   
   

   mysql> SELECT * from landscape\G 

   mysql> select count(1) from bucketactivitylog where destroy_time is null; 

   mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "security%"; 

   mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "alarm%"; 

   mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "ncm%"; 

   mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "modelcreatedestroybucket%"; 
   
   

4. Run the following command to stop and close the TEXT file: 
   

   mysql> \T 

5. Locate and upload the $SPECROOT/mysql/bin/queries.txt file 
   
   
6. Generate and upload the getinfo logs on the OC SRM. There is a utility called getSpectrumInfo.sh. You will find it under: 
   
   

   $SPECROOT/bin/support 

7. Run the following on the problematic server. 
   a. copy to your $SPECROOT folder. 
   
   b. open a bash shell. To open a linux bash shell on windows, from the run command, type 
   

   bash -login 

   
   c. Run:
   

   ./getSpectrumInfo.sh lite

   
   
   
8. what´s the disk usage?