DX UIM QOS data_engine Data Retention explained

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

How does Data Retention work in UIM? 

How to determine how much historic data is kept in the DB? 

 

How do Raw, Hourly Daily data is maintained in DX UIM? 

 

What do the Data Management Default Values mean? 

Are there any mechanisms available for archiving performance data within UIM events if the system were deleted to avoid monitoring alerts?

Environment

• DX UIM 20.4.x  / 23.4.x
• Data_engine 

Cause

Guidance / Data Management

Resolution

• The data_engine probe manages and maintains data that is collected from monitoring probes that emit Quality of Service (QoS) data.
  
  Data that is produced by monitoring probes is stored in the DX UIM database:
  
  The tables created in the UIM database have prefixes indicating the type of data they contain.
  
  S_ for tables that are used to store system data
  D_ for data tables
  H_ for tables containing historic data
  BN_ for data tables containing baseline data
  HN_ for data tables containing hourly/compressed data
  DN_ for data tables containing daily/compressed data
  RN_ for data tables containing unprocessed (raw) data directly from the probes
  
  
  
  
  
• Open data_engine GUI and configure the Data Management values that explained in detail below. 
  
  To enable data compression, Activate "Compress data before delete": If selected, compression is done on raw data and copied into historic tables before a delete is performed. And historic data is compressed and aggregated into Daily Data, before deleting it from the HN tables.
  
  
  
  
  
  
• Data maintenance GUI values / keys Explained: 
  
  
  
  

  Setting in data_engine GUI	key in data_engine.cfg	Description
  Compress data before delete	data_management_compress	When set to yes, this option causes the data from the RN tables to be summarized and stored in the HN tables (historic data.) If you disable data_management_compress by setting it to no, then the data will no longer be stored in the HN tables. In that case if still wanted to keep your QOS data longer you would have to increase the retention days for the raw data. If this option is selected, then by default, compression is done on raw data which is then copied into historic tables. Compression is only completed before a delete is performed. In addition, historic data is compressed and aggregated into Daily Data before deletion from the HN tables
  Delete raw data older than	delete_raw_samples	For how long single data points are kept in the UIM DB. Data is stored in the RN_* tables for the number of days defined in this parameter. Data that is older than the "delete_raw_samples" defined key is summarized and aggregated into Hourly data (HN_tables).
  Delete historic data older than	delete_history_samples	The delete_history_samples key determines for how long the data is kept in the HN_Tables. Data that is older than value defined in delete_history_samples gets aggregated into the  Daily data (DN_ tables).
  N/A	daily_avg_age	Determines how long daily aggregated values will be kept in DN_ tables. Daily table data older than the indicated number of days is deleted. See Raw Configure for this parameter
  N/A	daily_avg_age_extra	How many future partitions should be created for each DN table. On a partitioned DB, the data_engine needs a “current” partition where it can write data. That is the “today partition”. Each partition corresponds to a day. The “today partition” needs to be pre-allocated and ready to be written to each time the DE writes to the DB. When data maintenance runs, it creates the partition for the day. The problem is that data maintenance can fail, be stopped for some time (for whatever reason) or not complete exactly at the day boundary. For this reason, when it runs, it creates “extra” look-ahead partitions. These *_extra config fields determine how many future partitions should be created for each table.

  
  
  
  

Additional Information

Notes: 

• Granted, we know that some companies, due to the need for management reporting, or due to audits, regulations/HIPAA, or other business needs, e.g., the need to store data for longer periods of time, but if your tables become too large by saving too much raw/historical data, then data maintenance may slow down normal operations or portal performance if the tables grow too large. Table growth depends on several factors such as how many metrics are being monitored and how often (frequency). Best Practices for the UIM database assume a DBA/DBA Group is managing the UIM backend database. It also assumes that partitioning is enabled.
  
  
• Ideally, 30 and 180 are normally the optimal raw/historical settings but if you need to store more data for a longer period of time, and its mainly for reporting, a DB archival strategy (Data archiving is the practice of moving data that's no longer being used to a separate storage device) can be implemented for DX UIM, but there is currently no formal built-in archival function to 'offload' data from the UIM database for later access and reporting.
  
  
• Normally a solid DBA or the DBA Group will possess some collective knowledge and/or preferred approaches for archiving strategies, most often for performance or audit reporting. Some helpful information can be found online for the given database vendor/type, e.g., MS SQL Server,   Oracle, or MySQL
  
  
  

Related KB: 

For more details on data_engine raw config keys see the KB: data_engine raw configuration parameters explained