/nonconfig high on NSX managers due to IDS/IPS signature search index

Products

• VMware NSX

Issue/Introduction

• "Operations DB Disk Usage High" or "Operations DB Disk Usage Very High" alarm triggered on NSX Manager UI.
• IDS/IPS is enabled and the signature has been updated many times by internet or manual import.
• /nonconfig/search directory consumes large size. Check the directory size with root account using the cli command below.
• df -h output on Managers shows 10% on /nonconfig for all nodes which caused the alarm to rise. 

• root@nsx:~# df -h
  Filesystem                   Size  Used Avail Use% Mounted on
  udev                          24G     0   24G   0% /dev
  tmpfs                        4.8G  1.4M  4.8G   1% /run
  /dev/sda3                     11G  4.9G  4.9G  51% /
  tmpfs                         24G   52M   24G   1% /dev/shm
  tmpfs                        5.0M     0  5.0M   0% /run/lock
  tmpfs                         24G     0   24G   0% /sys/fs/cgroup
  /dev/sda1                    942M  7.1M  870M   1% /boot
  /dev/mapper/nsx-config__bak   29G  2.6G   25G  1% /config
  /dev/mapper/nsx-config        29G   22M   28G  1% /config_bak
  /dev/mapper/nsx-image         42G  1.8G   38G   5% /image
  /dev/mapper/nsx-repository    31G   11G   19G  36% /repository
  /dev/mapper/nsx-tmp          3.7G  2.6M  3.5G   1% /tmp
  /dev/mapper/nsx-var+dump     9.3G   24K  8.8G   1% /var/dump
  /dev/mapper/nsx-var+log       27G   11G   15G  43% /var/log
  /dev/mapper/nsx-secondary     98G   38M   93G   10% /nonconfig
  tmpfs                        4.8G     0  4.8G   0% /run/user/1007
  tmpfs                        4.8G     0  4.8G   0% /run/user/0

• /nonconfig/search directory consumes large size. Check the directory size with root account using the cli command below.
  

  du /nonconfig | sort -nr
  
  8958984 /nonconfig
  
  8272824 /nonconfig/search
  
  8272816 /nonconfig/search/nodes
  
  8272812 /nonconfig/search/nodes/0
  
  8272412 /nonconfig/search/nodes/0/indices
  
  7691520 /nonconfig/search/nodes/0/indices/H###################
  
  7691508 /nonconfig/search/nodes/0/indices/H###################/0
  
  7691480 /nonconfig/search/nodes/0/indices/H###################/0/index
  
  638552  /nonconfig/diskonlycorfutable

• Index of idssignature consumes gigabytes
  

  curl http://localhost:9200/_cat/indices | grep -i idssignature
  
  green open nsx_policy_idssignature                           H################### 1 0 5281096     0   7.2gb   7.2gb

 

Environment

VMware NSX 

Cause

Entities like IDS Signature are marked as candidates for delayed indexing. However, this also causes their deletions to be ignored. As a result, the index for IDSSignature continues to grow with each new bundle upload, while old signatures though deleted from Corfu ,still remain in the index.

 

Resolution

The issue is resolved in VMware NSX 4.2.1.

Workaround:

To reduce the size of the bloated index, execute

start search resync all

 In case only one of the NSX manager has  high usage for the /nonfig directory please execute this command on the affected Manager and see if the issue is resolved,  else execute command on all the NSX Managers one by one.

Please execute it on one node first. It will take couple of minutes depending upon the scale of the system. Once it is verified that UI is working fine after few minutes, the same procedure can be followed on other nodes as well.

Note: If your infrastructure contains a large number of hosts and edge nodes, the indexing process may take several hours. In such cases, please open a support request with Broadcom Support and refer to the relevant KB article to assist with deleting outdated index files.

For more information, see Creating and managing Broadcom support cases.

Additional Information

If this article did not help resolve your issue, you can review the following article for further reference: Troubleshooting disk space related issues on NSX Nodes