"Emergency expiration of data when disk usage reaches “Critical” level" is not purging data
Article ID: 279177
Updated On:
Products
Issue/Introduction
"Emergency expiration of data when disk usage reaches “Critical” level" is not purging data
Resolution
Please note that the design of the database expiration is to delete Reporter databases that's primarily marked as corrupt.
The purge process maintains storage limits by automatically removing older files when storage reaches a critical limit.
Purge initiates when disk usage approaches 1% of the disk usage critical limit. The purge operation is initiated by system configurations to trigger in the event of either reaching the configured time frame limit or upon reaching critical disk usage limits. Administrators have access to the purge configurations that determine when a purge initiates.
In addition to log source and database expiration parameters, a retention day count can be specified for each database, local log source, and the Cloud Log Download file directory.
Note 1: When overall disk usage is within one percent of the specified critical maximum, Reporter launches a process that examines the Reporter databases and deletes any that are marked Corrupt. If disk usage is still too high, this process repeatedly deletes the relatively oldest database data and/or processed access log files until disk usage is at least 1% below the critical maximum, or all purgeable data has been removed.
The oldest processed access log file is the file with the greatest difference between the log source's configured expiration date and the file's modification timestamp.
You can configure Reporter to purge when a disk capacity threshold is reached.
Case Scenario 1:
The scenario below illustrates the Emergency Purge process after enough data is purged to drop the disk usage limit into operational conditions.
Consider a log source called LS1 with this configuration:
LS1 - expire days: 40; retain days: 30; current oldest file: 38 days old
Assume that the critical disk usage limit is set to 80% by the user. When the disk usage reaches 79%, the Emergency Purge process is launched. The process identifies several corrupt databases in the system and purges them. After the removal of corrupt databases, the disk usage drops to 75%. Because the disk usage limit is below the configured threshold, the purge stops deleting files. The purge feature remains on but does not purge until the conditions (79% disk usage) are met again.
Case Scenario 2:
This scenario illustrates how aged data is purged relative to its expiration date, which can result in more recent data being deleted before older data. Deleting files from a specific log source or database does not remove files newer than the retain date of the respective resource.
Consider a log source that is called LS2 and a database that is called DB2, with this configuration:
LS2 - expire days: 60; retain days: 30; current oldest file: 56 days old
DB2 - expire days: 40; retain days: 20; current oldest data: 40 days old
Assume that critical disk usage limit is set to 96% and no databases are in a corrupt state. When the disk usage reaches 95%, the Emergency Purge process begins. The process identifies the 40-day-old data in DB2 as the relatively oldest (since its expire date minus data date is 0, which is greater than the –four day relative age of the oldest file in LS2). Assume that disk usage is still within 1% of the critical level after deleting that 40-day-old data from DB2. The process repeats, this time identifying and deleting the 39-day-old data from DB2. If necessary, this process repeats for 2 more iterations until the oldest DB2 data is 36 days old: the same expire relative age as the oldest file in the LS2 log source. If disk space still must be cleared, both the oldest file in LS2 and the oldest data in DB2 are deleted.
The two case scenarios are very unique, and are the only ways the "Purge Reporter Critical Disk Limits" is executed. Unless any of the conditions in both scenarios are met, no database or log data deletion would happen. So, even if you have configured data expiration to happen daily, if there are no corrupt database nor aged data that clearly meet the condition in the second scenario, no deletion would happen.
Now, because none of the above conditions are met, with reference to your configured settings, no deletion happens and the other option is to purge manually, based on the number of log lines.
Note 2: Most of the database is kept in memory. If the entire database is not occasionally purged, it continues to consume more of the process memory as new log files are processed. As the database grows, configuration settings that were previously beneficial might become detrimental.
As a general guideline, Symantec recommends that databases contain a maximum of 30 days of log data. However, the amount of log data (number of rows) has more impact than the number of days (age of data) in the data sets.
Feedback
