API Portal Legacy: Storing more or less than the default 60 days of metrics data.

Products

CA API Developer Portal CA API Gateway

Issue/Introduction

This article will speak to how to store more (or less) than the default 60 days of metrics data in the API Portal Legacy (previously known as Classic Portal after 4.0 was released - Classic Portal includes all versions 3.5 and lower).

The API Portal Legacy holds - by default - 60 days of metrics data. This can cause unintended behaviour though when generating API reports for a period of 6 months for example as it would only contain 2 months of data if left at the default value. Additionally if too much data is being consumed due to the metrics data this may need to be lowered from the default value. This article will discuss how to change that value.

Environment

API Portal Legacy 3.5 and lower.

Resolution

The Portal purges metrics data on a daily basis. The schedule is controlled by the Portal's Scheduler plugin.

Hourly metric data older than X days is removed and quarter hourly metric data older than 1 day is removed each time that the purger task is run. X is configurable using the PURGER_NUM_DAYS configuration property in the following file: /opt/Deployments/lrs/server/webapps/ROOT/plugins/lrsgateway-conf.xml

If a property named PURGER_NUM_DAYS is not present in the file, the default value of 60 is used which means the Portal will hold 60 days of metrics data.

If the metrics purge functionality is successful, the following log entries should be seen (as an example) at approximately 00:15 of every day:
05/15 00:15:00.007 INFO (SimpleThreadPoolWorker-8:) - Purging data with resolution=1 that is 60 day(s) older than Tue May 15 00:15:00 JST 2018 (period start less than or equal to 1521126900001) in batches of 100.
05/15 00:15:00.008 INFO (SimpleThreadPoolWorker-8:) - Finished purging data

If the PURGER_NUM_DAYS is missing in the file noted above, it can be added like so:

The following should be added in the <Properties> node of the lrsgateway-conf.xml file: <Property name="PURGER_NUM_DAYS">X</Property>
1. Change X to represent the number of days desired for keeping metrics data in the environment ("60" by default)
Restart the API Portal node. The restart is required for the new property to be recognized by the application.

Additional Information

Other useful information related to this topic:

A 15 minute metric is the hits for a 15 minute period (ex: 00:15 to 00:30). There are 96 of them in a day.
A 1 hour metric is the hits for a 60 period (ex: 00:00 to 01:00). There are 24 of them in a day.
These metrics are created by a cron job on the gateway.

Since, the 15 minute metrics are only used for the "Last 24 hours" report criteria any 15 minute metrics records older than yesterday are not necessary and are purged. For example: the purge runs at 00:00 on the 30th will purge records for 28th and keep the records for the 29th. Running a "Last 24 hours" report on the 30h when you select the from date as the 29th and the to date as the 30th will use the 15 minute metrics for the 29th and 30th.

It is not recommended to change the schedule of the 15 minute metrics purge since the purge removes records older than 24 hours. It needs to run at midnight so that it purges the correct data.