VMware NSX-T "Operations DB Disk Usage Very High" Alarm Triggered due to large search index
search cancel

VMware NSX-T "Operations DB Disk Usage Very High" Alarm Triggered due to large search index

book

Article ID: 371814

calendar_today

Updated On:

Products

VMware NSX Networking VMware NSX VMware NSX-T Data Center

Issue/Introduction

  • "Operations DB Disk Usage High" or "Operations DB Disk Usage Very High" alarm triggered on NSX Manager UI.
  • IDS/IPS is enabled and the signature has been updated many times by internet or manual import.
  • /nonconfig/search directory consumes large size. Check the directory size with root account using the cli command below.

    du /nonconfig | sort -nr
    8958984 /nonconfig
    8272824 /nonconfig/search
    8272816 /nonconfig/search/nodes
    8272812 /nonconfig/search/nodes/0
    8272412 /nonconfig/search/nodes/0/indices
    7691520 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw
    7691508 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0
    7691480 /nonconfig/search/nodes/0/indices/HWFhhx3bQfiJPAeQvgAkuw/0/index
    638552  /nonconfig/diskonlycorfutable

  • Index of idssignature consumes gigabytes.

    curl http://localhost:9200/_cat/indices | grep -i idssignature
    green open nsx_policy_idssignature                           HWFhhx3bQfiJPAeQvgAkuw 1 0 5281096     0   7.2gb   7.2gb

Environment

VMware NSX-T Data Center, VMware NSX

Cause

Deletion of IDSSignature is not processed by the search framework. Thus the search index gradually bloats on updating IDS/IPS signature.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

To reduce the size of the bloated index, execute

start search resync all

command on all the NSX Managers one by one.

Please execute it on one node first. It will take couple of minutes depending upon the scale of the system. Once it is verified that UI is working fine after few minutes, the same procedure can be followed on other nodes as well.

 

Additional Information