Large number of unprocessed modelcreatedestroybucket tables in SRM

Products

• Network Observability
• Spectrum

Issue/Introduction

We have a large number of modelcreatedestroybucket tables that are not finishing processing.

We launched the following query:

mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "modelcreatedestroybucket%";
+----------+
| count(1) |
+----------+
|     9512 |
+----------+
1 row in set (0.10 sec)

In the $SPECROOT/tomcat/logs/catalina.out file, the following error was found:

2025-05-13 16:40:31,695 [SRM/ModelCreateDestroyHandler5242880/bucketReader] WARN  com.aprisma.errorlog -  (SRM_Model_Create_Destroy_Events) - (ERROR) - Unknown exception encountered while processing name events:  processing halted for all servers

 

Caused by: org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [SELECT event_ID, time, event_seq, model_key, vardata, type, model_h, domain, event_key FROM modelcreatedestroybucket5242880_1744199926446 WHERE event_seq BETWEEN ? AND ?  ORDER BY event_seq]; SQL state [HY000]; error code [130]; Incorrect file format 'modelcreatedestroybucket5242880_1744199926446'; nested exception is java.sql.SQLException: Incorrect file format 'modelcreatedestroybucket5242880_1744199926446'

Cause

The problematic table sizes is 0 byte in the $SPECROOT/mysql/data/reporting/ directory:

-rw-r----- 1 spectrum spectrum      0 Apr  9 13:58 modelcreatedestroybucket5242880_1744199926446.MYD
-rw-r----- 1 spectrum spectrum      0 Apr  9 13:58 modelcreatedestroybucket5242880_1744199926446.MYI

Resolution

Delete the problematic modelcreatedestroybucket table.

1. Open a bash shell (bash -login)

2. Navigate to the $SPECROOT/mysql/bin/ directory.
cd mysql/bin

3. Launch the MySQL command prompt:
./mysql -uroot -p<password> reporting  (Windows)

./mysql --defaults-file=../my-spectrum.cnf -uroot -p<password> reporting   (Linux)

4. Run the following command to set the problematic bucket table has been processed in the bucketactivitylog table:

mysql> update bucketactivitylog set destroy_time = now() where bucket_table_name = 'modelcreatedestroybucket5242880_1744199926446';

5. Run the following command to drop the problematic bucket table from the MySQL reporting database:

mysql> DROP TABLE modelcreatedestroybucket5242880_1744199926446;

6. Quit from MySQL command prompt.

7. Delete the following files from the $SPECROOT/mysq/data/reporting/ directory (if they are still there):

-rw-r----- 1 spectrum spectrum      0 Apr  9 13:58 modelcreatedestroybucket5242880_1744199926446.MYD
-rw-r----- 1 spectrum spectrum      0 Apr  9 13:58 modelcreatedestroybucket5242880_1744199926446.MYI

8. Stop/start the Spectrum Tomcat service:

a) Stop the Spectrum Tomcat service.

ps -ef | grep tomcat | grep -v webtomcat | grep -v grep     (to list the running tomcat process)

$SPECROOT/tomcat/bin/stopTomcat.sh

b) Ensure the tomcat service is no longer running (it may take a couple of minutes):

ps -ef | grep tomcat | grep -v webtomcat | grep -v grep    (to list the running tomcat process)

If it is not stopping, it is hung, then kill it.

c) Start the Spectrum Tomcat service.

$SPECROOT/tomcat/bin/startTomcat.sh