NSX-T database encounters Out Of Memory errors and /config uses a lot of disk space when the number of Active Directory users exceeds the scale limit

Products

• VMware NSX

Issue/Introduction

• Manager status displays as UNKNOWN.
• NSX Manager when user runs df -h as root /config is constantly above 1%
• From the /var/log/corfu/tanuki.log file, you see entries similar to:
  
  INFO | jvm 14 | 2020/08/09 02:48:37 | Welcome to CORFU SERVER
INFO | jvm 14 | 2020/08/09 02:48:37 | Version (fd9ab60)
INFO | jvm 14 | 2020/08/09 02:48:37 | Serving on port 9000
INFO | jvm 14 | 2020/08/09 02:48:37 | Data location: /config
INFO | jvm 14 | 2020/08/09 03:59:32 | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | 2020/08/09 03:59:32 | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | 2020/08/09 03:59:32 | Dumping JVM state.
STATUS | wrapper | 2020/08/09 03:59:32 | The JVM has run out of memory. Restarting JVM.
INFO | jvm 14 | 2020/08/09 03:59:32 | Dumping heap to /image/core/corfu_oom.hprof ...
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 3.0.x
VMware NSX-T Data Center 3.1.x

Cause

This issue occurs when the system goes over scale limit of Active Directory users, with more than 100K DirectoryUser records in Corfu.

Note: VMware has published scale limit but NSX does not halt the system when limit has been broken.
Corfu, as of NSX-T Data Center 3.0, cannot support high storage usage per table.

Resolution

To resolve the issue, stay within the scale limits of Active Directory users for the installed NSX version.

Workaround:

To mitigate the current impact of being over the scale limit of Active Directory users, open a Service Request with Broadcom support to follow the workaround for this issue.