NSX-T alarm about disk usage for the Manager node disk partition /nonconfig has reached 10%

Products

• VMware NSX

Issue/Introduction

An alarm from NSX manager was raised such as the following:

 

The disk usage for the Manager node disk partition /nonconfig has reached 10% which is at or above the high threshold value of 10%. This can be an indication of rising disk usage by the NSX Datastore service under the /nonconfig/corfu directory.

Alert type: Problem
Severity: Warning
Manager: <NSX Manager IP>
Defined by: System
Alert tags: NSX-T
NSX-T manager: <NSX Manager IP>
Source Component: <NSX Manager Node>
NSX-T Alert Type: operations_db_disk_usage_high
NSX-T Alarm Status: OPEN
Created at: Jul 22, 04:45
Last Modified at: Jul 22, 04:45

Recommendation: Please run the following tool and contact GSS if any issues are reported /opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig

 

However, running the script errors out with the following error message:

Traceback (most recent call last):
  File "/opt/vmware/tools/support/inspect_checkpoint_issues.py", line 597, in <module>
    raise ValueError('A directory should be specified for the analysis to start.')
ValueError: A directory should be specified for the analysis to start.

 

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

Environment

VMware NSX-T 3.x and VMware NSX 4.x

Cause

The disk usage for the Manager node's disk partition /nonconfig has reached 10%, but the recommendation in the alarm provides an incomplete command.

Resolution

Issue has been resolved in NSX versions 3.2.4 and 4.2.0

A correct version of the command should be: 

For NSX-T 3.x: #/opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig --dir /var/log/corfu

For NSX 4.x: #/opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig --dir1 /var/log/corfu

However, this is a diagnostic tool which does not resolve the storage issue.

Workaround:

Alternatively run below Resync command on all 3 Managers as admin to resolve 

• SSH to all 3 NSX Managers
• Login as admin
• Run start search resync all on all 3 Managers
• Check alarm to see if it Resolves

If above Workaround doesn't Resolve alarm proceed with the following:

Please collect df -h and du -h /nonconfig output using root user and open a Support Request with Broadcom Support to further investigate the storage issue.

 

 

 

Additional Information