VMware NSX "Operations DB Disk Usage Very High" Alarm Triggered due to large search index

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• "Operations DB Disk Usage High" or "Operations DB Disk Usage Very High" alarm triggered on NSX Manager UI.
• IDS/IPS is enabled and the signature has been updated many times by internet or manual import.
• /nonconfig/search directory has high usage. Check the directory size with root account using the cli command below.
  
  du /nonconfig | sort -nr
8958984 /nonconfig
8272824 /nonconfig/search
8272816 /nonconfig/search/nodes
8272812 /nonconfig/search/nodes/0
8272412 /nonconfig/search/nodes/0/indices
7691520 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw
7691508 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0
7691480 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0/index
638552  /nonconfig/diskonlycorfutable
  
  
• Index of idssignature consumes gigabytes of disk space.
  
  curl http://localhost:9200/_cat/indices | grep -i idssignature
green open nsx_policy_idssignature                           HWFhhx3bQfiJPAeQvgAkuw 1 0 5281096     0   7.2gb   7.2gb

Environment

• VMware NSX-T Data Center
• VMware NSX
• VMware vDefend Firewall

Cause

Deletion of IDSSignature is not processed by the search framework. The search index gradually grows as the IDS/IPS signatures are updated.

Resolution

The issue is resolved in VMware NSX 4.2.1.

Workaround:

To reduce the size of the bloated index, execute the following command on all NSX manager nodes:

start search resync all

Additional Information

Related Known Issues:

• /nonconfig partition disk usage high and very high Alarm