The reporting database is out of sync. The SRM (Spectrum Report Manager) stopped processing the events.
mysql> select * from landscape\G
*************************** 1. row ***************************
landscape_h: 1048576
domain_name: #####
avail_sync_time: 2017-04-01 23:59:53
dev_sync_time: NULL
int_sync_time: NULL
process_if_app_events: 0
spm_sync_time: NULL
alarm_sync_time: 2017-04-01 23:59:40
event_sync_time: 2017-04-01 23:49:46
event_polling: 1
device_polling: 1
1 row in set (0.00 sec)
Release:
Component:
The following message errors were found in the Tomcat log file ($SPECROOT/tomcat/logs/stdout.log or catalina.out):
Apr 06, 2017 00:37:39.951 (SRM/LandscapeManager/LandscapeThread_0) (SRM_Events) - (ERROR) - Failed to insert event into event table. SQL Exception code: 1526 SQL Exception message: Table has no partition for value from column_list Last known event time = 1491101993000
Apr 06, 2017 00:37:39.951 (SRM/LandscapeManager/LandscapeThread_0) (SRM_Events) - (ERROR) - event record (mh=0x1233c2, type=0x10801, time=2017-04-02 00:00:01.0)
Apr 06, 2017 00:37:40.871 (SRM/LandscapeManager/LandscapeThread_0) (SRM_Events) - (ERROR) - Database ERROR processing events for domain #####. Will not poll domain
The partitioning handler was not triggered because the Tomcat service is being restarted every 24 hours.
Let the Spectrum Tomcat service running more than 24 hours, so the partition handler will be triggered and the new event partition tables will be created.
In the tomcat log file we can see the event tables were created.
abr 11, 2017 12:40:02 AM(INFO)Partition Handling Task started...
abr 11, 2017 12:40:02 AM(INFO)In getMissingPartitions method...
abr 11, 2017 12:40:02 AM(INFO) The last partition name is : EVENT_P91
abr 11, 2017 12:40:02 AM(INFO) We need to create 6 number of partition(s).
abr 11, 2017 12:40:02 AM(INFO) Missing Partition(s) created successfully.
abr 11, 2017 12:40:02 AM(INFO) Updating Partitionlog table.
abr 11, 2017 12:40:02 AM(INFO) Partitionlog table updated with 6 entries.
abr 11, 2017 12:40:02 AM(INFO)In deleteOldPartitions method...
abr 11, 2017 12:40:02 AM(INFO)8 number of partition(s) should be deleted.
abr 11, 2017 12:40:02 AM(INFO) Deletion of older partitions completed successfully.
The Spectrum Tomcat service needs to be recycled to process the events again.
Or you can change when the partition handler is triggered.
1.Navigate to $SPECROOT\tomcat\webapps\spectrum\WEB-INF\repmgr\config\container
2.Edit the Scheduling.xml file.
3.Search for partitionHandlerScheduler bean, change the start delay from 300000 (5 minutes) to 3600000 (1 hour).
<bean id="partitionHandlerScheduler"
class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean">
<property name="jobDetail" ref="partitionHandlerTask" />
<property name="startDelay" value="300000" /> [Change this value to 3600000]
<property name="repeatInterval" value="86400000" />
</bean>
4.Save the file and restart the tomcat.
The change will now trigger the partition handler after one hour of tomcat restart instead of default 5 minutes. The repeat interval still holds true with repeat every one day.