Troubleshooting Spectrum Failed OLB, hung, paused, Online Backup Model alarm

Products

CA Spectrum DX NetOps

Issue/Introduction

Troubleshooting Spectrum Online Backup

Environment

Release: All Spectrum Versions

Component: SPCCSS - SpectroSERVER Core

Resolution

Check Online Backup model for events and alarms:

1. In OneClick, go to Locator Search
2. search By Application Models > By Name
3. search "online" to locate all Online Backup models in your DSS environment
4. select the OLB model having issue and review events and alarms

The Spectrum Online Backup fails with the following two Events/Alarms on the OnLineBackup model:

OnLineBackup (name - OnLineBackup) has reported a general failure!.

If an Online Backup has failed previously, the tmp file created may be left, preventing a new backup attempt.

1. Check for a left over "tmpdir" directory in the location where the backup files are saved during an Online Backup process.
2. Remove any left over "tmpdir" directories and try another backup

Hanging Online Backup:

Delete the temp directory for the backup from $SPECROOT/SS-DB-Backup
Go to the OnLineBackup model in OneClick
Go to the Attributes tab
Change the following:
1. BackupNow to "No"
2. BackupStatus to "Ready"
3. PauseVNM to Yes

You should now see "Begin Backup Now" and "Ready" under the VNM model > Online Database Backup. This should allow you to run an OnlineBackup. If it hangs again please open a support ticket.

Online Backup Fails with Error : Failed To Open Destination File:

There are several reasons why this could occur:

When there is a temporary directory (*tmpdir) left over from a previous backup in the $SPECROOT/SS-DB-Backup directory (default), or the directory configured by the user to save the backup to.
When there is not enough space available to save the file(s).
Permissions on the backup directory have been changed.
On Windows, if there is another rm.exe program installed on the system besides the one in the $SPECROOT/NT-Tools/SRE/bin directory

To solve:

Check the contents of the backup directory for any left over tmpdir directory and remove
Ensure there is available space
Check and verify the permission on the backup directory
Remove any rm.exe programs other than the one in the $SPECROOT/NT-Tools/SRE/bin directory

Enable Online Backup Debugging:

1. Within OneClick, go to VNM model of the same landscape where the Online Backup issue exists
2. Go to Information > Dynamic Debugging
3. Enable Online Backup debug

The debug outputs to <SPECROOT>/SS/VNM.OUT

You can also enable debug via command line

Fault Tolerant Database Sync Failure:

Review the following log files for errors.

on Primary server:

<SPECROOT>/lib/SDPM/processd_log
<SPECROOT>/SS/RCPD.OUT

on Secondary server:

<SPECROOT>/lib/SDPM/processd_log
<SPECROOT>/SS/RCPD.OUT

Database Sync occurs on port 51966 - 0xCAFE - by default. Make sure this is open in local and network firewalls, or change in /SS/.vnmrc: rcpd_comm_port=<port number in hex>

Additional Information

There may be times when the SpectroSERVER threads are consumed causing latency issues. This may prevent the online backup from running after changing the attributes. If you try to stop the SpectroSERVER, you are unable to stop it. In this instance you will need to kill the SpectroSERVER and reload a previously saved copy from the last time it successfully ran.

Hanging online backup

Fault Tolerant Backups take a long time