"Manager disk usage high" alarm reported on NSX Manager nodes.

Products

• VMware NSX

Issue/Introduction

• Below alarm is reported on all or any NSX manager nodes.

• Alarm triggered on the NSX Manager node indicates a high disk usage on the /repository partition:
  • "The disk usage for the Manager node disk partition /repository has reached 86%, which is at or above the high threshold value of 80%."
• Upon further investigation of the disk usage on the affected NSX Manager nodes, the following disk usage is observed:

Filesystem                   Size  Used  Avail  Use% Mounted on
/dev/sda2                     11G   7.4G   3.6G   67% /
/dev/mapper/nsx-repository    31G  26.7G   4.3G   86% /repository

• The root partition (/) is within acceptable usage limits at 67%. 
• However, the /repository partition is at 86% usage, which exceeds the configured high threshold (typically 80%) leading to the alarm getting triggered.
• Checking further on the contents of the repository directory, it was observed that the older NSX version image file was still present on the node, triggering the alert.

ls -ltrh /repository/
total 24K
drwxrwx--- 2 uuc grepodir  16K MMM DD HH:MM lost+found
drwxrwx--- 7 uuc grepodir 4.0K MMM DD HH:MM <Old_NSX_Version_image>
drwxrwx--- 7 uuc grepodir 4.0K MMM DD HH:MM <Current_NSX_Version_image>
-rwxrw---- 1 uuc grepodir   19 MMM DD HH:MM current_version

Environment

VMware NSX

Cause

The "/repository" directory on the NSX Manager stores files like NSX upgrade bundles, patch files and installation images which are used during system upgrades and patching workflows. 

If these files are not cleaned up after upgrades, they accumulate over time and lead to the consumption of significant disk space.

Resolution

• First validate what is the current version of NSX.
• It can be checked from the About page on NSX UI or it can also be done by running the below command from Admin shell on NSX manager node: 
  • get version
• After validating the version, please remove the old image file.
• Once the file has been removed, the alarm will automatically resolve or it can be resolved manually by going to the NSX Home page -> Alarms -> Select the alarm -> Resolve.

Additional Information

For more information regarding manager disk usage high alarms, please refer: Manager disk usage high alarm