Global Manager "Manager config disk usage High" Alarm for /config
search cancel

Global Manager "Manager config disk usage High" Alarm for /config

book

Article ID: 381334

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Global Manager or Local Manager showing alarms for Manager Config Disk Usage High.

  • Description: The disk usage for the Manager node disk partition /config has reached 10% which is at or above the threshold value of 10%. This can be an indication of rising disk usage by the NSX Datastore service under the /config/corfu directory

df -h output on Global Managers shows 10% on /config for all nodes which caused the alarm to rise. There is no impact on the environment, and /config size does not increase.

root@nsx:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                          24G     0   24G   0% /dev
tmpfs                        4.8G  1.4M  4.8G   1% /run
/dev/sda3                     11G  4.9G  4.9G  51% /
tmpfs                         24G   52M   24G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                         24G     0   24G   0% /sys/fs/cgroup
/dev/sda1                    942M  7.1M  870M   1% /boot
/dev/mapper/nsx-config__bak   29G  2.6G   25G  10% /config
/dev/mapper/nsx-config        29G   22M   28G   1% /config_bak
/dev/mapper/nsx-image         42G  1.8G   38G   5% /image
/dev/mapper/nsx-repository    31G   11G   19G  36% /repository
/dev/mapper/nsx-tmp          3.7G  2.6M  3.5G   1% /tmp
/dev/mapper/nsx-var+dump     9.3G   24K  8.8G   1% /var/dump
/dev/mapper/nsx-var+log       27G   11G   15G  43% /var/log
/dev/mapper/nsx-secondary     98G   38M   93G   1% /nonconfig
tmpfs                        4.8G     0  4.8G   0% /run/user/1007
tmpfs                        4.8G     0  4.8G   0% /run/user/0

Environment

VMware NSX 4.x

Cause

This issue can be hit on any NSX environment that has upgraded from 3.x to 4.x 

 

/config/search/ folder contains year-old indices, which were used in previous versions of NSX before we migrated from Elasticsearch to OpenSearch.

 

root@nsxmgr:~# ls -lrt /config/search/nodes/0/indices
total 328
drwxr-x--- 4 elasticsearch elasticsearch 4096 Nov  3  2023 klTt1cdkRemtqLpcmDRU5Q
drwxr-x--- 4 elasticsearch elasticsearch 4096 Nov  3  2023 jziX8TDKT66M58PR_U5Sow
drwxr-x--- 4 elasticsearch elasticsearch 4096 Nov  3  2023 Yh4a18QaSr2XOGqbTD_ZfA
drwxr-x--- 4 elasticsearch elasticsearch 4096 Nov  3  2023 Hjgix_-HTVuQWag8qoJ0vg
drwxr-x--- 4 elasticsearch elasticsearch 4096 Nov  3  2023 fk1sR9pvRlC_hFOLAGWpXg

 

 

It is safe to delete /config/search folder as the current installation uses /nonconfig/search/ folder and we confirmed with output of command "ps -ef | grep search.conf" which contains "-Epath.data=/nonconfig/search" as seen below:

 

root@nsxmgr:~# ps -ef | grep search.conf
nsx-sea+ 84863 84684 0 Jun25 ? 16:27:06 /usr/lib/jvm/openjdk-java8-runtime-amd64/bin/java -Djava.awt.headless=true -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -Xloggc:/var/log/search/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djava.security.manager -Djava.security.policy=/etc/search/conf/search.policy -Dlogging.subcomp=search -Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/etc/opensearch -XX:+UseStringDeduplication -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/image/core/search_oom.hprof -Djava.io.tmpdir=/nonconfig/search/tmp -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:ErrorFile=/var/log/search/hs_err.log -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Dlog4j2.asyncLoggerRingBufferSize=65536 -Xms1925m -Xmx1925m -Djava.library.path=/usr/tanuki/bin/../lib -classpath /usr/tanuki/bin/../lib/wrapper.jar:/usr/share/opensearch/lib/HdrHistogram-2.1.9.jar:/usr/share/opensearch/lib/hppc-0.8.1.jar:/usr/share/opensearch/lib/jackson-core-2.12.5.jar:/usr/share/opensearch/lib/jackson-dataformat-cbor-2.12.5.jar:/usr/share/opensearch/lib/jackson-dataformat-smile-2.12.5.jar:/usr/share/opensearch/lib/jackson-dataformat-yaml-2.12.5.jar:/usr/share/opensearch/lib/java-version-checker-1.2.3.jar:/usr/share/opensearch/lib/jna-5.5.0.jar:/usr/share/opensearch/lib/joda-time-2.10.4.jar:/usr/share/opensearch/lib/jopt-simple-5.0.2.jar:/usr/share/opensearch/lib/jts-core-1.15.0.jar:/usr/share/opensearch/lib/log4j-api-2.17.1.jar:/usr/share/opensearch/lib/log4j-core-2.17.1.jar:/usr/share/opensearch/lib/lucene-analyzers-common-8.10.1.jar:/usr/share/opensearch/lib/lucene-backward-codecs-8.10.1.jar:/usr/share/opensearch/lib/lucene-core-8.10.1.jar:/usr/share/opensearch/lib/lucene-grouping-8.10.1.jar:/usr/share/opensearch/lib/lucene-highlighter-8.10.1.jar:/usr/share/opensearch/lib/lucene-join-8.10.1.jar:/usr/share/opensearch/lib/lucene-memory-8.10.1.jar:/usr/share/opensearch/lib/lucene-misc-8.10.1.jar:/usr/share/opensearch/lib/lucene-queries-8.10.1.jar:/usr/share/opensearch/lib/lucene-queryparser-8.10.1.jar:/usr/share/opensearch/lib/lucene-sandbox-8.10.1.jar:/usr/share/opensearch/lib/lucene-spatial3d-8.10.1.jar:/usr/share/opensearch/lib/lucene-spatial-extras-8.10.1.jar:/usr/share/opensearch/lib/lucene-suggest-8.10.1.jar:/usr/share/opensearch/lib/nsx-opensearch-1.0.jar:/usr/share/opensearch/lib/opensearch-1.2.3.jar:/usr/share/opensearch/lib/opensearch-cli-1.2.3.jar:/usr/share/opensearch/lib/opensearch-core-1.2.3.jar:/usr/share/opensearch/lib/opensearch-geo-1.2.3.jar:/usr/share/opensearch/lib/opensearch-launchers-1.2.3.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-1.2.3.jar:/usr/share/opensearch/lib/opensearch-secure-sm-1.2.3.jar:/usr/share/opensearch/lib/opensearch-x-content-1.2.3.jar:/usr/share/opensearch/lib/snakeyaml-1.26.jar:/usr/share/opensearch/lib/spatial4j-0.7.jar:/usr/share/opensearch/lib/t-digest-3.2.jar -Dwrapper.key=xNmwFFT4e6BJHnUSW40Q_spZniWFDCRe -Dwrapper.port=32004 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=84684 -Dwrapper.version=3.5.41-pro -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=/usr/tanuki/bin/../lang org.tanukisoftware.wrapper.WrapperSimpleApp com.vmware.nsx.service.Main --daemonize -Enode.name=nsx_node -Epath.logs=/var/log/search -Epath.data=/nonconfig/search -Eindices.query.bool.max_clause_count=30000 -Ecluster.routing.allocation.disk.watermark.low=25mb -Ecluster.routing.allocation.disk.watermark.high=25mb -Ecluster.routing.allocation.disk.watermark.flood_stage=10mb -Ecluster.routing.allocation.disk.include_relocations=false -Elogger.index.search.slowlog=info -Elogger.index.indexing.slowlog=info -Eclient.transport.ignore_cluster_name=true -Esearch.max_buckets=20000

 

 

Resolution

Step 1: Verify if there are old indices stored in /config/search

  • Run command ls -lrt /config/search/nodes/0/indices on Manager nodes as root to confirm old indices

Step 2: Confirm that the current NSX installation is utilizing /nonconfig/search folder

  • Run command ps -ef | grep search.conf on Manager nodes as root to confirm the output contains "-Epath.data=/nonconfig/search"

Step 3: Delete /config/search folder from each Manager that matches above criteria

  • Delete /config/search folder by running the command rm -rf /config/search/ as root on each Manager node.

Step 4: Confirm that all /config directories now show single digit percentage on all nodes.

  • Run command df -h as root user to confirm /config folder is healthy and has decreased to single digit percentage

 

If the above criteria and workaround do not apply please open a case with VMware by Broadcom Support.

Powered by Wolken Software