NSX-T alarm about disk usage for the Manager node disk partition /nonconfig has reached 10%

Products

• VMware NSX

Issue/Introduction

• Below alarm from NSX manager was raised :
  

  The disk usage for the Manager node disk partition /nonconfig has reached 10% which is at or above the high threshold value of 10%. This can be an indication of rising disk usage by the NSX Datastore service under the /nonconfig/corfu directory.
  
  Alert type: Problem
  Severity: Warning
  Manager: <NSX Manager IP>
  Defined by: System
  Alert tags: NSX-T
  NSX-T manager: <NSX Manager IP>
  Source Component: <NSX Manager Node>
  NSX-T Alert Type: operations_db_disk_usage_high
  NSX-T Alarm Status: OPEN
  
  Recommendation: Please run the following tool and contact GSS if any issues are reported /opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig

• However, running the script [/opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig] errors out with the following error message: 
  

  Traceback (most recent call last):
    File "/opt/vmware/tools/support/inspect_checkpoint_issues.py", line 597, in <module>
      raise ValueError('A directory should be specified for the analysis to start.')
  ValueError: A directory should be specified for the analysis to start.

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The disk usage for the NSX Manager node's disk partition /nonconfig has reached 10%, but the recommendation in the alarm provides an incomplete command.

Resolution

Issue has been resolved in below NSX versions.
VMware NSX-T Data Center 3.2.4 and later
VMware NSX 4.2.0 and later

The correct version of the command should be: 

For NSX-T 3.x : /opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig --dir /var/log/corfu
For NSX 4.x   : /opt/vmware/tools/support/inspect_checkpoint_issues.py --nonconfig --dir /var/log/corfu

However, this is a diagnostic tool which does not resolve the storage issue.

Workaround:

Alternatively run below Resync command on all 3 Managers as admin to resolve 

• SSH to all 3 NSX Managers
• Login as admin
• Run from admin mode  on all 3 Managers start search resync all
• Check alarm to see if it Resolves

If above Workaround doesn't Resolve alarm proceed with the following:

Please collect df -h and du -h /nonconfig output using root user and open a Support Request with Broadcom Support to further investigate the storage issue.

Additional Information

Related Known Issues:

• /nonconfig partition disk usage high and very high Alarm