When the mysqlbackup fails during the execution of post_olb_script, there is no event and alarm generated for the OnLineBackup (OLB).
The following alarm was expected:
The mysqlbackup is failing with errors that are logged in the $SPECROOT/SS-DB-Backup/ddm_backup.log file, but the $retval variable is returning 0 (successful), hence there is no error in the scheduled OnLineBackup (OLB).
Starting database save to file: /opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.mbi
MySQL Enterprise Backup version 4.1.2 Linux-4.1.12-37.4.1.el6uek.x86_64-x86_64 [2018/07/23 15:13:55]
Copyright (c) 2003, 2018, Oracle and/or its affiliates. All Rights Reserved.
[Build ID : 13401143.32e59573fc6d0520d268d395dda7c9fd8b5427f5]
201231 11:20:25 MAIN INFO: A thread created with Id '139908347578176'
201231 11:20:25 MAIN INFO: Starting with following command line ...
--compress --user=spectrum --password=xxxxxxxx
--include-tables=^ddmdb. --use-tts backup-to-image
201231 11:20:25 MAIN INFO:
201231 11:20:25 MAIN INFO: Not using SSL.
201231 11:20:25 MAIN INFO: MySQL server version is '5.7.27-enterprise-commercial-advanced-log'
201231 11:20:25 MAIN INFO: MySQL server compile os version is 'linux-glibc2.12'
201231 11:20:25 MAIN INFO: Got some server configuration information from running server.
201231 11:20:25 MAIN INFO: Server system variable 'old_alter_table' was set to '0'. Setting it to '1'.
201231 11:20:25 MAIN INFO: Skipping binlogs and relaylogs in case of TTS backup.
IMPORTANT: Please check that mysqlbackup run completes successfully.
At the end of a successful 'backup-to-image' run mysqlbackup
prints "mysqlbackup completed OK!".
201231 11:20:25 MAIN INFO: MEB logfile created at /opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.tmpdir/meta/MEB_2020-12-31.11-20-25_compress_img_backup.log
201231 11:20:28 PCR1 ERROR: Page at offset 50331648 in /opt/application/spectrum/mysql/data/ddmdb/event#P#EVENT_10_12_2020.ibd seems corrupt!
mysqlbackup failed with errors!
201231 11:20:28 MAIN INFO: Setting server system variable 'old_alter_table' back to '0'.
We are not able to capture the error_code/exit_code from the ddm_save.pl script and pass to the post_olb_script which can detect the error and raises the OLB failure event and alarm.
Release : 10.4.1
Component : Spectrum Alarm & Event Management
The following scripts need to be modified:
In the $SPECROOT/SS/DDM/ddm_save.pl file:
Replace the lines 94 and 95 with the following entries:
Replace the lines 174 to 177 with the following entries:
exit($exit_code >> 8);
In the $SPECROOT/custom/CsScript/post_olb_script file - add the following entries on line 80 (Note this change is not mandatory, it is just to print the error code in the ddm_backup.log file):
if [ retval -eq 0]
echo "ddm_save.pl/mysqlbackup is Successful !!!" >> $backup_dir/ddm_backup.log
echo "ddm_save.pl/mysqlbackup is failed with an error code : $retval" >> $backup_dir/ddm_backup.log
The changes in the scripts will be delivered in the post 21.2.2 Spectrum release.
Also, check this KB article:
ddm_save.pl does not zip the save file enough (poor/low compression on the DDMdb backup file)
Uncomment out the following lines to run DDMd backup on a weekly basis, on Sundays:
Uncomment out the following lines to run DDMd backup on a daily basis: