My NFA Harvester Server is running out of Disk space.
Article ID: 46447
Updated On:
Products
Issue/Introduction
My NFA Harvester is running out of available Disk Space and I am afraid the drive will fill up.
***
NOTE: If you are running NFA 21.2.10, make sure you have the mandatory Data Retention patch installed:
NFA 21.2.10 Drive filling Patch | UPDATED 5/23 (broadcom.com)
***
The NFA Harvester is responsible for processing and storing a high volume of data.
In a 2 tier environment a Harvester is responsible for storing three types of data:
1. Flow Forensics Data (Stored for up to 24-48 hours) stored in the \<Install Dir>\Netflow\datafiles\HarvesterArchive\ directory.
2. 1 Minute Resolution Data (Stored for up to 30 days) stored in the \<Install Dir>\Netflow\datafiles\ReaperArchive\ directory.
3. 15 Minute Resolution Data (Some of which is stored for up to 13 months) stored in the \<Install Dir>\Netflow\datafiles\ReaperArchive15\ directory.
Data is pruned and maintained in two ways by the "CA NFA Data Retention" service:
1. Daily - This data types mentioned above are pruned automatically back to its preset limits daily if the server has sufficient space by the "CA NFA Data Retention" service.
2. Threshold Based -The "CA NFA Data Retention" service also will always try to ensure that you have at least 10% free disk space (5% in earlier versions) at all times. The "CA NFA Data Retention" service will check the server once a minute to see if the 10% free disk space threshold has been violated. If so it will prune data back, in order to maintain the 10% free disk space.
There are a few situations in which Data Pruning may fail and may cause you to run out of disk space.
1. If the "CA NFA Data Retention" service is down data pruning will stop.
2. If the "CA NFA Reaper" service is down, files will start to build up in the \<Install Dir>\Netflow\datafiles\ReaperInput and \<Install Dir>\Netflow\datafiles\ReaperWork\ directories being that data cannot be processed.
Being that the "CA NFA Data Retention" service does not prune the ReaperInput/ReaperWork directories, the Data retention service will keep pruning the directories that it can and eventually prune all of your data. Eventually that will cause the Data Retention Service to stop, and the the ReaperInput\ReaperWork directories to fill up the rest of the drive.
3. The Data Retention service is taking too long to calculate the size and number of folders of the ReaperArchive15 directories.
Environment
NFA 9.1 and later
Resolution
1. Ensure that your "CA NFA Data Retention" service and the "CA NFA Reaper Service" are running at all times.
2. Ensure you have configured your Harvester with the correct amount of Storage according to the NFA System Requirements to ensure you have enough storage for the amount of flows and interfaces you are monitoring with each Harvester.
3. Check the dataretention-wrapper.log from the \CA\NFA\Netflow\logs directory, and search for "Refreshing datastore HistoricalArchive".
Check the timestamp for the line that says "Refreshing datastore HistoricalArchive" and check the timestamp of the next line after that.
Under normal circumstances, this process should only take a few minutes or less.
If you see that the difference in the two timestamps is taking a long time, it could be why your Data Retention service is not pruning data fast enough.
INFO | jvm 1 | 2019/03/07 19:21:39 | 7:21:39 PM - [INFO ] - Refreshing datastore HistoricalArchive
INFO | jvm 1 | 2019/03/07 21:28:12 | 9:28:12 PM - [INFO ] - As datastore Start Time : 1546128000
INFO | jvm 1 | 2019/03/07 21:28:12 | 9:28:12 PM - [INFO ] - As datastore End Time : 1552003200
INFO | jvm 1 | 2019/03/07 21:28:12 | 9:28:12 PM - [INFO ] - As datastore Current Size In Bytes: 2506847736
When the Data Retention service prunes data, it leaves behind empty folders in the \CA\NFA\Netflow\datafiles\ReaperArchive15 directory. Over time the number of these empty folders add up and and can cause the Data Retention service to be too slow to prune data.
There is a Powershell script that can be run to delete these empty folders.
The command must be run in a powershell window on the affected Harvester server.
Be sure to cd to the parent directory containing the empty directories you want to delete on the Harvester.
<install_path>\Netflow\datafiles\ReaperArchive15
Make sure Powershell 3.0 or 4.0 is installed. If you need to upgrade the Powershell version, you should also install the appropriate MySQL connector for Powershell 3.0+:
https://dev.mysql.com/get/Downloads/Connector-Net/mysql-connector-net-6.9.9.msi
The following powershell command will remove empty directories in the working directory:
dir -Directory -Recurse | where {($_ | dir).Count -eq 0} | rmdir
Feedback
