In NAPP environments, the 'rawflowcorrelator' and 'overflowcorrelator' pods can sometimes malfunction, potentially leading to indefinite growth in disk usage and causing failures in log bundle collection processes.
search cancel

In NAPP environments, the 'rawflowcorrelator' and 'overflowcorrelator' pods can sometimes malfunction, potentially leading to indefinite growth in disk usage and causing failures in log bundle collection processes.

book

Article ID: 319824

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

From the 'df -h' command output, it's evident that the  '/var/log/napp/supportbundle' directory on the 'cluster-api' pod is almost full or full. To address this issue, a patch named "cluster_api_patch.yaml" is suggested. This patch is to be applied to the 'cluster-api'. The steps to apply this patch are provided in 'Workaround Section'.

Symptoms:
1. Alarms for platform services disk usage growing.
2. Failure to generate a support bundle.

Relevant Log Locations

The issue can be confirmed by collecting the "diskusage.txt" file after executing the following commands:
napp-k exec -it `napp-k get pods | grep cluster | cut -d ' ' -f 1` -c cluster-api -- find /var/log/napp/supportbundle -type f -exec ls -lh {} \; > ./diskusage.txt

napp-k exec -it `napp-k get pods | grep cluster | cut -d ' ' -f 1` -c cluster-api -- df -h >> ./diskusage.txt


Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 4.x

Cause

In certain scenarios, the 'rawflowcorrelator' and 'overflowcorrelator' pods can sometimes malfunction, leading to excessive and uncontrolled log generation. This abnormal behavior results in rapid disk space consumption, potentially causing indefinite growth in disk usage. This issue is particularly severe if these pods crash or restart frequently, exacerbating log accumulation and posing a risk to system performance and storage capacity.

Resolution

This is now fixed in NAPP 4.1.2 and higher versions
NAPP 4.1.2 do not need to patch the setup.

The patch "cluster_api_patch" attached to this KB has been tested on NAPP 4.1.1 release version and after applying the patch , flowcorelator files older than 2 days were deleted.

Workaround:
Download the attached file "cluster_api.yaml" and move it to the NSX manager node.
Execute the following command on the NSX shell from the same directory where the file is present:
napp-k patch deployment cluster-api --type merge --patch-file=cluster_api_patch.yaml


Additional Information

Impact/Risks:
Alarms will be seen about platform services disk usage, and it will not be possible to generate the NAPP support bundle.
It is a known issue in 4.1.x and lower versions.

Attachments

cluster_api_patch get_app