NSX manager UI becomes inaccessible due to /config partition filling up rapidly.

Products

• VMware NSX

Issue/Introduction

• Users are unable to log in to the NSX UI (via VIP or specific node IPs).
• UI displays the error:
  Some appliance components are not functioning properly. Component health: SEARCH:DOWN, MANAGER:DOWN, NODE_MGMT:UP, UI:UP. Error code: 101


• High disk usage observed on the /config partition (e.g., jumping from 72% to 85%).
• In /var/log/cbm/tanuki.log, recurring OOM errors are observed:
  The JVM has run out of memory. Requesting thread dump.

• In /var/log/corfu/corfu-compactor-audit.log, the compactor fails during checkpointing:
  java.lang.OutOfMemoryError: Java heap space (also generating a heap dump at /image/core/compactor_oom.hprof).
  
  
• ProfileMsg table dump analysis reveals an abnormally high number of entries (e.g., ~185K entries) related to latency profiles.

Environment

VMware NSX

Cause

This issue occurs because specific Corfu tables, primarily ProfileMsg and LatencyStatProfile, grow excessively large due to a stale profile issue originating from VMware Aria Operations for Networks.
When the Corfu compactor attempts to process these severely bloated tables, it exceeds its allocated memory and crashes. Because the compactor cannot successfully complete its cycle, the database is never compacted, and the /config directory continues to fill up.

Resolution

This issue is resolved in VMware NSX 4.2.2, available via Broadcom downloads.

WORKAROUND

If an immediate upgrade is not feasible, apply the following workaround:

1. Deactivate the NSX management cluster and shrink it down to a single manager node.

2. Temporarily increase the compactor memory limits by editing /usr/share/corfu/scripts/compactor_runner.py.

3. Run manual compaction and trim to reduce the /config partition size and stabilize the node.

4. Revert the compactor memory configuration back to default (300m) and redeploy two fresh manager nodes to restore the cluster.

ADDITIONAL INFORMATION

• For assistance with software access, review the Download Broadcom products and software KB article.

• If you are unable to upgrade and the workaround fails, review the Creating and managing Broadcom support cases. KB article and reference this specific issue when opening a case.

 

Additional Information

NSX Manager Data Collection Failure in AON/vRNI Due to Latency Profile Deletion Error