VMware NSX-T "Operations DB Disk Usage Very High" Alarm Triggered due to large search index

Products

• VMware NSX

Issue/Introduction

• "Operations DB Disk Usage High" or "Operations DB Disk Usage Very High" alarm triggered on NSX Manager UI.
• IDS/IPS is enabled and the signature has been updated many times by internet or manual import.
• /nonconfig/search directory consumes large size. Check the directory size with root account using the cli command below.
  
  du /nonconfig | sort -nr
8958984 /nonconfig
8272824 /nonconfig/search
8272816 /nonconfig/search/nodes
8272812 /nonconfig/search/nodes/0
8272412 /nonconfig/search/nodes/0/indices
7691520 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw
7691508 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0
7691480 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0/index
638552  /nonconfig/diskonlycorfutable
  
  
• Index of idssignature consumes gigabytes.
  
  curl http://localhost:9200/_cat/indices | grep -i idssignature
green open nsx_policy_idssignature                           HWFhhx3bQfiJPAeQvgAkuw 1 0 5281096     0   7.2gb   7.2gb
  
  

Environment

VMware NSX-T Data Center, VMware NSX

Cause

Deletion of IDSSignature is not processed by the search framework. Thus the search index gradually bloats on updating IDS/IPS signature.

Resolution

The issue is resolved in VMware NSX 4.2.1.

Workaround:

To reduce the size of the bloated index, execute

start search resync all

command on all the NSX Managers one by one.

Please execute it on one node first. It will take couple of minutes depending upon the scale of the system. Once it is verified that UI is working fine after few minutes, the same procedure can be followed on other nodes as well.

 

Additional Information

Also, please check the following KB: 'Operations Db Disk Usage High' or 'Operations DB Disk Usage Very High' alarms are triggered