Spectrum OnLineBackup Failing - "FAILURE DURING COMPRESSION OF THE SAVEFILE"

Products

• Network Observability
• Spectrum

Issue/Introduction

Spectrum OnLineBackup is failing when trying to compress backup file.

Alarm Title: FAILURE DURING COMPRESSION OF THE SAVEFILE
Event: OnLineBackup (name - OnLineBackup): The compression of the database savefile failed during an On-line backup.

After turning on Spectrum Debug: https://knowledge.broadcom.com/external/article/49821/how-to-enable-debug-for-spectrum-online.html and reproducing, seeing the below in VNM.OUT:

gzip return bad exit code - 13

Oct 16 11:47:20 OLB TRACE at CsIHOB.cc(2248): Compressing database

Oct 16 11:47:22 OLB TRACE at CsIHOB.cc(2286): gzip return bad exit code - 13

Oct 16 11:47:22 OLB TRACE at CsIHOB.cc(2308): Done compressing database - 6

Oct 16 11:47:22 OLB TRACE at CsIHOB.cc(1898): Done copying the database - 6

Oct 16 11:47:22 OLB TRACE at CsIHOB.cc(883): One of the OLB attributes(BackupNow, NextBackupDateTime, AutomaticBackups) change has triggered OLB. 

Oct 16 11:47:22 OLB TRACE at CsIHOB.cc(883): One of the OLB attributes(BackupNow, NextBackupDateTime, AutomaticBackups) change has triggered OLB. 

Oct 16 11:47:24 OLB TRACE at CsIHOB.cc(3902): Done performing backup

Typically points to a permission issue or disk space issue.

Cause

Permission issue of $SPECROOT/Install-Tools/gzip or $SPECROOT/SS-DB-Backup folder

Spectrum owner account does not have ability to execute gzip or write to SS-DB-Backup folder

Resolution

Check Permissions and Owner of gzip included with Spectrum:
cd $SPECROOT/Install-Tools/
ls -lah | grep -i zip

Run gzip command manually:
touch /tmp/testfile.txt
cd $SPECROOT/Install-Tools/
gzip /tmp/testfile.txt > $SPECROOT/SS-DB-Backup

Check Permissions and Owner of Backup Directory:
cd $SPECROOT
ls -lah | grep -i SS-DB-Backup

Check Directory listing of Backup Directory for any temp files to be removed:
cd $SPECROOT/SS-DB-Backup
ls -lah

Check space of Backup Directory:
df -h $SPECROOT/SS-DB-Backup

Run chown and or chmod commands as needed to make changes to ownership and permission to files. They are most likely owned by root and need to be changed back to spectrum owner

Additional Information

If owner is changed to root, there may be a bigger issue here. Files that should be owned spectrum owner have been taken over by root

Resolution: Reinstall of Spectrum with correct owner in hostargs or a changing owner of directory structure may needed.