Alerts are not generated for alert configuration events
Article ID: 400559
Updated On:
Products
Issue/Introduction
Alerts are not generated for events from any alert configuration, this affects both existing and new alert configurations for any system event.
In this example, there were se_cpu_high events that occurred but the no alerts are found in Operations > Alerts > All Alerts
In the alert manger logs, you will find that alert objects stopped being created and more recent events did not create any corresponding alerts.
Log location: /var/lib/avi/log/alert_notifications_debug*log*
/var/lib/avi/log]
└─$ zgrep -ih 'Created' alert_notifications_debug*log* | sort | tail -n 5
[2025-05-04 04:22:19,261] INFO [alert_engine.processEvent:974] Created new Alert Instance for id: gslbservice-uuid
[2025-05-04 04:22:19,293] INFO [alert_engine.processEvent:974] Created new Alert Instance for id: gslbservice-uuid
[2025-05-04 04:22:19,328] INFO [alert_engine.processEvent:974] Created new Alert Instance for id: gslbservice-uuid
[2025-05-04 04:22:19,366] INFO [alert_engine.processEvent:974] Created new Alert Instance for id: gslbservice-uuid
[2025-05-04 04:22:19,404] INFO [alert_engine.processEvent:974] Created new Alert Instance for id: gslbservice-uuidAnother sympton is replay of alerts for older events.
Log location:
/var/lib/avi/log/alert_notifications_debug*log*
The timestamp on the alert that is being raised (Unix timestamp 1739595840) if converted is from 02.15.2025 @ 05:04:00 UTC while the alert being raised has timestamp 05.04.2025 @ 04:22:19 UTC. This is confirmed a replay of alerts for older events.
Timestamp conversion from the event:
admin@example:~$ date -d @1739595840
Sat 15 Feb 2025 05:04:00 AM UTCEvent triggering the alert:
[2025-05-04 04:22:19,765] INFO [alert_manager.raiseAlertTask:1153] obj: gslbservice-uuid acfg alertconfig-uuid finished raising alert:
[2025-05-04 04:22:19,765] INFO [alert_manager.raiseAlertCallback:1199] Alert alertconfig-uuid for gslbservice-uuid Alert UUID
[2025-05-04 04:22:19,773] INFO [alert_manager.raiseAlertTask:1007] updated alertperf_obj
[2025-05-04 04:22:19,774] INFO [alert_manager.raiseAlertTask:1152] nosavealert:uuid: ""
alert_config_uuid: "alertconfig-uuid"
timestamp: 1739595840.0
obj_uuid: "gslbservice-uuid"
threshold: 0
events {
report_timestamp: 1739595840
obj_type: GSLBSERVICE
event_id: SE_EV_GS_MEMBER_UP
module: SE_HM
internal: EVENT_EXTERNAL
context: EVENT_CONTEXT_SYSTEM/var/lib/avi/log/event_manager*INFO*
This issue can also be identified in the event manager logs, the events from the alert manger logs can be correlated (alert_notifications_debug*log*)
2025-05-04T04:22:19.047Z E 12473 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057887742, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.351Z E 12473 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895643, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.451Z E 12473 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895653, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.552Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895660, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.553Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895671, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.553Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895679, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.553Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895688, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.553Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895698, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.553Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895707, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.
2025-05-04T04:22:19.554Z E 4360 eventmanager/event_manager_streamer.go:215 Event with ReportTimestamp 7471507241057895715, event_id SE_EV_GS_MEMBER_UP and obj type GSLBSERVICE delayed by more than 6.736768e+06 seconds.This issue can also occur due to the system not being able to process the event "DEBUG_MODE_ON". This is natively found in the alert configuration "System-SE-Alert"
Environment
Affects Versions:
30.1.1
30.1.2 - 30.1.2-2p3
30.2.1 - 30.2.1-2p6
30.2.2 - 30.2.2-2p2
31.1.1 - 31.1.1-2p3
Cause
This is caused by a known issue, where the alert maneger is not able to process the event "DEBUG_MODE_ON" and alert replay of old events.
Resolution
Please upgrade/patch the system to the fix version to recover the system and resolve this issue.
ID: AV-226165
Description Alerts are not being generated on toggling the option SE DEBUG_MODE.
Fix Version: 30.2.2-2p3, 30.2.3
There are no workarounds available to recover the system from this issue.
Feedback
