How does the new Data Retention Service work? How can the Data Retention settings be changed?

Products

• Network Flow Analysis

Issue/Introduction

By default, the Data Retention settings are set to:

• Raw .NFA (HarvesterArchive) data files saved for 24 hours maximum and then pruned once per day
• 1 Minute (ReaperArchive) data files saved for 30 days and then pruned.
• 15 Minute (ReaperArchive15) data files set to 12 months and then pruned.

How can these settings be adjusted?

Environment

Any NFA release

Resolution

For information on the Data Retention service, please see the following page in the NFA Documentation:

Data Retention (broadcom.com)

 

In order to adjust the Data Retention settings, please see the following:

**Disclaimer** Please note, increasing data retention can cause problems with server performance so it is recommended to stay with the default levels if possible. The default settings are what were tested by QA on the server specs documented in the Installation Guide**

The new pruning service checks are run every minute to determine if there is a need to start cleaning out data (to prevent the disk from being fully consumed), but the pruner only runs once a day to dial back down to the "max" threshold (in the 'data_retention.datastores' table). So for some time periods of the day, you can see more than 24 hours' worth of NFA files.

 

 

**Note** 86400=24 hours. The amount of time is stored in seconds.

type_id 1= Harvester: ReaperArchive15; 15 minute data files

type_id 2= Harvester: ReaperArchive; 1 minute data

type_id 3= Harvester: HarvesterArchive; raw NetFlow data files use in Flow Forensics reports.

 

An example for changing the Retention service to retain HarvesterArchive raw Netflow data for 48 hours as opposed to 24 hours is as follows:

            Log into MySql from a Windows or Linux Command Prompt using the command below on either the Harvester:

            mysql -P3308 -unetqos -pnetqos data_retention

When you get the welcome to MySql message do a 'Select' query first to make verify the current configuration.

The 'Select' query is as follows:

         select * from datastores;

 

Once you verify the current settings you can start the 'Update' query.

For example to set the data retention for NFA files to 48 hours use the update statement below:

         update datastores set maxThreshold='172800' where name='NFA';

Confirm by using the 'Select' query again from step 2.

Recycle the "CA NFA Data Retention" service on the Harvester for the changes to take effect.

Confirm by using the 'Select' query again from step 2.

Reserved Disk Space Configuration and Options

The new service is also responsible for more aggressive pruning if the available free space on the drive falls to less than 10%. (10% is the default settings for reserved space on the harvester)

In the data_retention.settings table we can adjust the reserved hard drive space.

 

Right now (by default) the value for freeSpacePercentage = '10'. If changed to '15', the drive will reserve 15% as opposed to 10%

Be careful changing these settings as you could lose 15 or 1 minute data.

An example for changing the Retention service to reserve 15% of the disk as opposed to 10% is as follows:

Log into MySql from a Windows or Linux Command Prompt using the command below on the Harvester:

         mysql -P3308 -unetqos -pnetqos data_retention

When you get the welcome to MySql message do a 'Select' query first to make verify the current configuration.

        select * from settings;

Once you verify the current settings you can start the 'Update' query.

For example to update the database to change the reserved disk space to 15% you can use the update statement below:

         update settings set value='15' where name='freeSpacePercentage';

Confirm by using the 'Select' query again from step 2.

Recycle the "CA NFA Data Retention" service on the Harvester.

 

Data Retention Priority

The Data Retention Service also has a much more broad way on controlling what data gets cleaned out first. This can be viewed here in the data_retention.types table.

 

 

The 'Priority' Column is what we will be looking at here. The higher the number, the lower the priority so essentially in the default instance above, 'NetflowArchive' will start cleaning out first.

This table is not recommended to be altered.