Failure in the mysqlbackup during the execution of post_olb_script does not generate event and alarm

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

When the mysqlbackup fails during the execution of post_olb_script, there is no event and alarm generated for the OnLineBackup (OLB).


The following alarm was expected:

The mysqlbackup is failing with errors that are logged in the $SPECROOT/SS-DB-Backup/ddm_backup.log file, but the $retval variable is returning 0 (successful), hence there is no error in the scheduled OnLineBackup (OLB).

Starting database save to file: /opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.mbi
MySQL Enterprise Backup version 4.1.2 Linux-4.1.12-37.4.1.el6uek.x86_64-x86_64 [2018/07/23 15:13:55] 
Copyright (c) 2003, 2018, Oracle and/or its affiliates. All Rights Reserved.
[Build ID : 13401143.32e59573fc6d0520d268d395dda7c9fd8b5427f5]

201231 11:20:25 MAIN    INFO: A thread created with Id '139908347578176' 
201231 11:20:25 MAIN    INFO: Starting with following command line ...
 /opt/application/spectrum/SS/DDM/mysqlbackup 
        --defaults-file=/opt/application/spectrum/mysql/my-spectrum.cnf 
        --compress --user=spectrum --password=xxxxxxxx 
        --backup-dir=/opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.tmpdir 
        --backup-image=/opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.mbi 
        --include-tables=^ddmdb. --use-tts backup-to-image 

201231 11:20:25 MAIN    INFO: 
201231 11:20:25 MAIN    INFO: Not using SSL.
201231 11:20:25 MAIN    INFO: MySQL server version is '5.7.27-enterprise-commercial-advanced-log'
201231 11:20:25 MAIN    INFO: MySQL server compile os version is 'linux-glibc2.12'
201231 11:20:25 MAIN    INFO: Got some server configuration information from running server.

201231 11:20:25 MAIN    INFO: Server system variable 'old_alter_table' was set to '0'. Setting it to '1'.
201231 11:20:25 MAIN    INFO: Skipping binlogs and relaylogs in case of TTS backup.
IMPORTANT: Please check that mysqlbackup run completes successfully.
           At the end of a successful 'backup-to-image' run mysqlbackup
           prints "mysqlbackup completed OK!".

201231 11:20:25 MAIN    INFO: MEB logfile created at /opt/application/spectrum/database_sldclr0335_backup/db_20201231_1119.DDMDb.tmpdir/meta/MEB_2020-12-31.11-20-25_compress_img_backup.log
.
.
.
201231 11:20:28 PCR1 ERROR: Page at offset 50331648 in /opt/application/spectrum/mysql/data/ddmdb/event#P#EVENT_10_12_2020.ibd seems corrupt!

mysqlbackup failed with errors!
201231 11:20:28 MAIN    INFO: Setting server system variable 'old_alter_table' back to '0'.

 

Environment

Release : 10.4.1

Component : Spectrum Alarm & Event Management

Cause

We are not able to capture the error_code/exit_code from the ddm_save.pl script and pass to the post_olb_script which can detect the error and raises the OLB failure event and alarm.

Resolution

The following scripts need to be modified:

• $SPECROOT/SS/DDM/ddm_save.pl
• $SPECROOT/custom/CsScript/post_olb_script (change on this script is not mandatory)

 

In the $SPECROOT/SS/DDM/ddm_save.pl file:

Replace the lines 94 and 95 with the following entries:

    my $return_value=ddm_save();
    exit($return_value);

Replace the lines 174 to 177 with the following entries:

   my $exit_code=system($cmd);
 
   cleanup();
   exit($exit_code >> 8);

 

In the $SPECROOT/custom/CsScript/post_olb_script file - add the following entries on line 80 (Note this change is not mandatory, it is just to print the error code in the ddm_backup.log file):

 if [ retval -eq 0]
 then
  echo "ddm_save.pl/mysqlbackup is Successful !!!" >> $backup_dir/ddm_backup.log
 else
  echo "ddm_save.pl/mysqlbackup is failed with an error code : $retval" >> $backup_dir/ddm_backup.log
 fi

 

The changes in the scripts will be delivered in the post 21.2.2 Spectrum release.

Additional Information

Also, check this KB article:

ddm_save.pl does not zip the save file enough (poor/low compression on the DDMdb backup file)
https://knowledge.broadcom.com/external/article?articleId=199272

 

Uncomment out the following lines to run DDMd backup on a weekly basis, on Sundays:

 

Uncomment out the following lines to run DDMd backup on a daily basis: