NSX-T Edge Node High Disk Space Consumption

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are using IDPS on North South connections.
• When you check the disk space on the edge node, using df -h, you see the mount point /mnt/ids is using a large amount of space and does not reduce over time.
• Checking logs in /var/log/syslog* on the edge node, you can see similar messages:

Notification : { <payload sent by IDS Engine> } will not be processed.SH-IDS channel is not up. <Number of notifications> notifications are already queued.Thus rejecting any further notifications.

• On the edge node as root, if you run the command docker ps, you see the Security hub container is not running and the Datapathd service container may also not be running.

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

The Security hub container is not running. When this container is not running, files extracted by IDPS do not get removed and can lead to disk space growing.

Resolution

This is a known issue impacting NSX-T data center.

Workaround:
Delete stale files from /mnt/ids folder, however some files might be in the process of getting analysed and we only want to delete the stale files.
Follow the steps mentioned below to delete stale files.

1. Check /mnt/ids folder on the edge node. If this folder does not contain any files, ignore the following steps.
2. If /mnt/ids folder contains files, grep for each file in /mnt/ids in /var/log/syslog* using the following command :

grep "<file_name_in_/mnt/ids_folder>" /var/log/syslog*

2.a. If you do find this file in syslog, with the string "Security hub inspection event to sa-event-processor service", then this file is in use and not stale, do not delete these files.
Delete any other files not found as above 2.a. and restart the required container/services noted above as not running:
For example:

docker restart service_security_hub
docker restart service_datapath