/config partition disk usage high and very high Alarm

Products

• VMware NSX

Issue/Introduction

Title: Alarm for /config partition disk usage high and very high
Event ID: manager_health.manager_config_disk_usage_high, manager_health.manager_config_disk_usage_very_high

Alarm Description

• Purpose: Corfu database disk usage reaches limits
• Impact: This might be due to temporary heavy workload in NSX manager and the alarm will be cleared automatically after a few hours. If the high disk usage persists for a few days, Corfu database might experience clustering instability and performance degradation. If the disk usage keeps growing, the database will turn to read-only mode and further configurations in NSX manager will fail.

Environment

VMware NSX-T Data Center

Resolution

Steps to Resolve

For 3.0.0 and higher

Resolution:

For 4.1.1 and lower, keep monitoring until disk usage goes down automatically and the alarm is cleared in the next a few hours.

For 4.1.2 and later, run CorfuServer ODS Runbook from CLI to get more insight into underlying infra and Corfu compactor health. Note that this command might take up to 5 mins to return, and it might fail if SHA is not in a healthy state.

    manager1> start invocation runbook CorfuServer

Note: ODS Runbook above needs to be run as admin on 1 of the managers in the cluster.

Look for the result of step 4. The following example shows that Compactor is working fine.

Step Number : 4
Step Action : Check trim token movement in the given time window (default is 24h)
Step Result : The result of the Corfu Trim Token Movement Check is {'result': <Result.SUCCESS: 'SUCCESS'>, 'message': 'Detected a successful log trim.', 'data': {'last_trim_date': '2024-04-18T19:02:40.256Z'}}

If the alarm persists for a few days, or the ODS Runbook complaints about trim token not moving, contact VMware Support.

Maintenance window required for remediation? Yes