Oracle Database Failed to Startup

Products

• Data Loss Prevention
• Data Loss Prevention Core Package
• Data Loss Prevention Enforce
• Data Loss Prevention Enterprise Suite
• Data Loss Prevention Oracle Standard Edition 2

Issue/Introduction

Issue may be seen initially as one of the following...
- Console is unable to load.
- Enforce Services will not start or stay started.
- Unable to start the Oracle Database
- Unable to communicate with the Oracle Database

 

When you attempt to manually startup the Oracle Database you see the following error...
- ORA-03113: end-of-file on communication channel

If you check the Oracle Alert logs you may see the following items(C:\oracle\diag\rdbms\protect\protect\trace)...
- ORA-19809: limit exceeded for recovery files
- ORA-19804: cannot reclaim 104857600 bytes disk space from 80530636800 bytes limit

Environment

Applies to all versions of Oracle and DLP

Cause

This issue is caused by the use of "ArchiveLogMode".
- Archive Log Mode is not supported by Broadcom
- This issue occurs because Archive Log Mode has filled up the available space and can no longer generate new logs.

Resolution

First, lets confirm Archive Log Mode is enabled.
- Log into your Oracle Database as sysdba
- Run the command "archive log list;"

This shows you if Archive Log Mode is enabled, where the log files are located, and the current sequence

At this point you have 3 options, only one of which is supported by Broadcom...
OPTION# 1 (Supported, permanent): Disable Archive Log Mode and manually cleanup the files. When you need to perform Backups, you will want to perform a Cold Backup as described in our Oracle Maintenance Guide. This method prevents any files from being created in the first place and is the only permanent solution to this issue.

OPTION# 2 (UNSupported, Temporary): Run RMAN to perform your backup, this will cleanup all of the files that have currently built up. Note that this MUST be performed regularly or the files will build up and result in your Database shutting down again.

OPTION# 3 (UNSupported, Temporary): Increase the available memory for "db_recovery_file_dest_size". You can see the current value of this by using the "Show parameter db_recovery_file" command, then look for the db_recovery_file_dest_size in the list. This will show you the current amount of disk space allocated to Archive storage. If you have available disk space you can increase this  value then restart the Database at which point it should immediately come online. However, please note that this is a temporary fix. At some point you will need to perform one of the previous options to cleanup the space that is consumed, or the Database will simply consume the newly available space in a few days and then you will be right back where you started.