How can I get the required maximum storage size for the Data Repository?
Article ID: 43645
Updated On:
Products
Issue/Introduction
This article introduces method to calculate CAPM Data Repository storage size as capacity planning from actual historical data.
Environment
PM 20.x or above
Cause
Would it be possible to anticipate required maximum storage size for Data Repository?
Resolution
Use the CAPM Sizing Tool is the primary source for checking required maximum storage size.
This article introduces the method of estimating the maximum storage size of Data Repository which the current monitoring setting will be operating continuously.
1) Analyze the Data Repository storage size by tables
Run the SQL: select * from projection_storage;
Example: In this case, top 3 ifstats tables occupies 92% of total DR storage size.
|
table_name |
MB |
|
ifstats_rate |
2333.349 |
|
ifstats_hourly_baselines |
383.0694 |
|
ifstats_ltd |
299.1711 |
2) Calculate current Rollup cover rate
Example “ifstats_rate” table
SQL: select item_id, count(*) from ifstats_rate group by item_id order by item_id;
|
A |
B |
C |
D |
E |
|
number of lines of each item_id accumulated in table?descending order? |
Number of Item_id |
Polling interval: max lines in table (*Note1) |
{A}/{C} |
{B}*{D} |
|
64869 |
1 |
1 min: 64800 |
1.001065 |
1.001065 |
|
34018 |
24 |
1 min: 64800 |
0.524969 |
12.59926 |
|
34017 |
147 |
1 min: 64800 |
0.524954 |
77.16819 |
|
34016 |
5 |
1 min: 64800 |
0.524938 |
2.624691 |
|
34015 |
15 |
1 min: 64800 |
0.524923 |
7.873843 |
|
34013 |
1 |
1 min: 64800 |
0.524892 |
0.524892 |
|
12976 |
1 |
5 min: 12960 |
1.001235 |
1.001235 |
|
6805 |
35 |
5 min: 12960 |
0.525077 |
18.3777 |
|
6804 |
88 |
5 min: 12960 |
0.525 |
46.2 |
|
6803 |
1 |
5 min: 12960 |
0.524923 |
0.524923 |
|
6802 |
5 |
5 min: 12960 |
0.524846 |
2.624228 |
|
Total |
323 |
|
170.52 |
Current rollup accumulation rate of ifstats_rate table = sum{E} / sum{B} = 170.52 / 323 = 0.527926 = 53%
Current table’s physical disk size is 2333.349 MB. Then full accumulation of rollup will reaches = 2333.349 MB / 53% = 4419.843 MB.
Possibility disk usage size will require extra = 4419.843 – 2333.349 = 2086.494 MB
3) The top 3 tables occupies 92% of total DR storage size are ifstats_rate, ifstats_hourly_baseline and ifstats_ltd in this system.
Following is the extra required storage size of other tables:
- ifstats_hourly_baseline: 338.1678 MB
- ifstats_ltd: 1260.359 MB
Sum up those 3 tables extra required size is 2086.494 + 338.1678 + 1260.359 = 3085.0208 MB.
Current DU command result of /data and /catalog directory is 5867MB.
Conclusion, it will grow the storage with full Rollup as 3085.0208 + 5867 = 9552.0208MB.
*Note1: The maximum table lines accumulated by rollup processing
- Polled data: 45 days => RATE tables: [1 minutes interval] 60*24*45=64800, [5 minutes interval] 12*24*45=12960
- Hourly rollup data: 90 days => LTD tables: 24*90=2160?
- Daily rollup data: 365 days?=> EQD tables: 365
- Weekly rollup data: 730 days=> GTD tables: 730/7=?104
Additional Information
For example: You need to increase the Data Retention to 90 days:
To get a rough idea of current 45 day as-polled usage, in the vertica vsql, you can run:
select node_name, sum(used_bytes) from projection_storage where projection_schema='dauser' and projection_name like '%_rate_%' group by 1;
That will show number of bytes per node for all as-polled (rate) tables. Double it at for a base number for 90 days. Then a % for future item growth.
ltd (hourly rollup)
eqd (daily rollup)
gtd (weekly rollup)
select node_name, sum(used_bytes) from projection_storage where projection_schema='dauser' and projection_name like '%_ltd_%' group by 1;
select node_name, sum(used_bytes) from projection_storage where projection_schema='dauser' and projection_name like '%_eqd_%' group by 1;
select node_name, sum(used_bytes) from projection_storage where projection_schema='dauser' and projection_name like '%_gtd_%' group by 1;
Attachments
Feedback
