DX NetOps Spectrum Online Backup failing

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

Backup failing in primary MLS.

Online backup model failing in primary MLS.

We see an Alarm with the following details. (values not useful for this article are removed, not full Alarm shown)

Severity : Critical
Name : OnLineBackup
Type : OnLineBackup
Acknowledged : Yes
Alarm Title : FAILURE DUE TO SPECTRUM OR THE DATABASE BEING LOCKED
Landscape : <Name> (0x1000000)
Event : OnLineBackup (name - OnLineBackup) has reported a general failure!.
Source : SPECTRUM
Change Owner : SPECTRUM
Symptoms : SPECTRUM and/or the database has become locked.
Probable Cause :

1) The backup sub-system could not read the present directory from the system.  This may occur if the directory where the SpectroSERVER is running is removed.
2) The database could not be paused/closed.  The database close failure may occur if the database lock file ".VNMDB.LOCK" has been removed or modified.
3) A database open failure occurred after a successful close.

Actions :

1) Run "pwd" from the command line in the SS directory and see if it works correctly.  If it does not, shut down and re-start the SpectroSERVER.
2) Check for the existence of a file named ".VNMDB.LOCK" in the SS directory.  If it is not there, shut down and re-start the SpectroSERVER.

Environment

All supported DX NetOps Spectrum releases

Cause

There are multiple problems that can cause this.

1. $SPECROOT/SS/.VNMDB.LOCK file is:
   • Not present.
   • Not owned by the install owner.
   • Sample from a Support lab:
     • [root@<SS_HostName> ~]# ls -al /usr/Spectrum/SS/.VNMDB.LOCK
       -rw-r--r-- 1 spectrum spectrum 82 Jun 30 00:00 /usr/Spectrum/SS/.VNMDB.LOCK
2. $SPECROOT/SS/SS-DB-Backup directory is not owned by or writable by the install owner.
   1. Sample from a Support lab:
      • [root@<SS_HostName> ~]# ls -al /usr/Spectrum/ | grep SS-DB-Backup
        drwxr-xr-x   2 spectrum spectrum      4096 Jun 30 00:00 SS-DB-Backup
3. $SPECROOT/SS/SS-DB-Backup directory contains a leftover *.tmpdir directory from a failed backup run.

Resolution

To resolve this take the following steps.

1. What was the cause?
   • Missing .VNMDB.LOCK file? Stop and restart the SpectroSERVER.
   • Wrong SS-DB-Backup owner or rights/perms? Correct them.
   • Leftover *.tmpdir? Delete the directory and it's contents.
2. Run a new backup.
   • Go to the VNM model, select the Component Detail pane, expand the Online Database Backup section. Run a backup.
   • Confirm it completes successfully.

Additional Information

Does the VNM model show a button that us not available for backup runs? Does the backup seem to be hung in the UI? To reset it follow the steps in this KB article.

Troubleshooting Spectrum Failed OLB, hung, paused, Online Backup Model alarm