AIOps - What unnecessary logs and files can I delete?

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

DX OI nfs filesystem is almost full "87%". 

/dev/mapper/rootvg-lvvarNFSdoi  837G  690G  111G  87% /var/nfs/doi

When running du -shc <nfs-oi-folder>/* | sort -h, we can see below directories are using the most disk space

Can I delete these logs?

 

Environment

DX Operational Intelligence 1.3.x, 20.2
DX Application Performance Management 11.x, 20.2
App Experience Analytics (AXA) 17.3.x

Resolution

If AIOps 20.x

Delete old log files (*.log.gz"):

cd <NFS directory>

- List all archived log files:

find . | grep "log\.gz"  

or in addition calculate the total size:

find . -type f -name '*log\.gz' -exec du -ch {} +

Here is an example of the output:

4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-27-11-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-18-11-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-27-17-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-18-17-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-28-05-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-18-23-1.log.gz
4.0K    ./jarvis/verifier/logs/jarvis-verifier-7848b65b4-584b2/verifier.2021-02-19-05-1.log.gz
...

IMPORTANT: make sure to delete only the files that end in ".log.gz"

- Review the output, it should list only archived log files, then delete all *log.gz:

cd <NFS directory>

find . -type f -name '*log\.gz' -exec rm -rf {} +

 

If DX OI 1.3.2

Option 1: Reduce DSP trans logs

cd <NFS directory>/dsp-logs
rm -rf ./trans.log.<year>*

Option 2: Delete DSP Green Highways raw data

cd <NFS directory>/dsp-model-data/dsp_green_highways
rm -rf ./dsp_gh_data_*

Option 3. Delete old log files (*.log.gz"):

To list all the archived logs, run:

cd <NFS directory>

- List all archived log files:

find . | grep "log\.gz"  

or in addition calculate the total size:

find . -type f -name '*log\.gz' -exec du -ch {} +

Here is an example of the output:

./normalized-alarm-logs/archive/normalized_alarm_2020-09-25_1.log.gz
./normalized-alarm-logs/archive/normalized_alarm_2020-09-13_1.log.gz
./normalized-alarm-logs/archive/normalized_alarm_2020-09-29_2.log.gz
./normalized-alarm-logs/archive/normalized_alarm_2020-08-26_5.log.gz
./normalized-alarm-logs/archive/normalized_alarm_2020-10-11_1.log.gz
..

./jarvis/apis-logs/apis.2020-06-23.log.gz
./jarvis/apis-logs/apis-onboard.2020-09-29.log.gz
./jarvis/apis-logs/apis.2020-09-03.log.gz
./jarvis/apis-logs/apis-onboard.2020-10-17.log.gz
./jarvis/apis-logs/apis.2020-10-21.log.gz
./jarvis/apis-logs/apis-onboard.2020-08-1

./jarvis/verifier-logs/verifier.2020-07-28-11.log.gz
./jarvis/verifier-logs/verifier.2020-02-11-07.log.gz
./jarvis/verifier-logs/verifier.2020-09-03-21.log.gz
./jarvis/verifier-logs/verifier.2020-04-09-21.log.gz
..
./jarvis/utils-logs/jarvis-es-utils.2020-04-22-11.log.gz
./jarvis/utils-logs/jarvis-es-utils-Rollover.2020-08-19-01.log.gz
./jarvis/utils-logs/jarvis-es-utils-Purge.2020-03-30-21.log.gz
./jarvis/utils-logs/jarvis-es-utils.2020-07-13-07.log.gz
..

IMPORTANT: make sure to delete only the files that end in ".log.gz"

- Review the output, it should list only archived log files, then delete all *log.gz:

cd <NFS directory>

find . -type f -name '*log\.gz' -exec rm -rf {} +

 

For AXA 17.3.2

Delete old log files (*.log.gz"):

cd <NFS directory> or <AXA-HOME>

- List all archived log files:

find . | grep "log\.gz"  

or in addition calculate the total size:

find . -type f -name '*log\.gz' -exec du -ch {} +

IMPORTANT: make sure to delete only the files that end in ".log.gz"

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html