Manager Config Disk Usage Very High Alarm with ProfileMsg Corfu Failures
search cancel

Manager Config Disk Usage Very High Alarm with ProfileMsg Corfu Failures

book

Article ID: 419046

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX UI alarm stating that Manager config disk usage is very high

  • NSX cluster is stable and all services are running.
  • VRNI is or was previously used within the environment.
  • Config partition is >15% and <50% usage. This can be checked in appliances page on the NSX GUI (NSX GUI > System > Appliances or by cli using get filesystem-stats as admin.
  • Alarms present for core dumps on Managers, /image/core directory confirms the presence of core dump files, such as compactor_oom.hprof and cbm_oom.hprof. 
  • On one of the NSX manager nodes, login as root and run the command grep -B5 'Dumping heap' /var/log/corfu/corfu-compactor-audit.log  to confirm log lines similar to below, the table being read during the OOM state is nsx$ProfileMsg :

    2026-05-11T07:40:05.416Z | INFO  |              Cmpt-chkpter-9000 |     o.c.runtime.view.SMRObject | ObjectBuilder: open Corfu stream nsx$ProfileMsg id b810b3e3-####-####-####-78c38cc330a6
    2026-05-11T07:40:05.450Z | INFO  |              Cmpt-chkpter-9000 |  o.c.r.c.PersistedStreamingMap | Cleared RocksDB data on /config/corfu-compactor/compactor_nsx_ProfileMsg
    2026-05-11T07:40:05.483Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: Started checkpoint for b810b3e3-####-####-####-78c38cc330a6 at snapshot Token(epoch=4, sequence=4085767)
    2026-05-11T07:40:05.559Z | INFO  |              Cmpt-chkpter-9000 |  o.c.r.c.PersistedStreamingMap | Cleared RocksDB data on /config/corfu-compactor/compactor_nsx_ProfileMsg
    java.lang.OutOfMemoryError: Java heap space
    Dumping heap to /image/core/compactor_oom.hprof ...
  • /var/log/corfu-compactor-leader log, may contain logs similar to the below showing a failure due to a lack of checkpoint activity for the same table (ProfileMsg):

    2026-05-12T09:56:35.589Z | WARN  |              Cmpt-9000-chkpter |               compactor-leader | Marked table nsx$ProfileMsg FAILED due to no checkpoint activity
    2026-05-12T09:56:35.589Z | INFO  |              Cmpt-9000-chkpter |               compactor-leader | Invoking finishCompactionCycle
    2026-05-12T09:56:35.604Z | INFO  |              Cmpt-9000-chkpter |               compactor-leader | Total time taken for the compaction cycle: 466390ms for 1087 tables with status FAILED

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This issue occurs because specific Corfu tables, primarily ProfileMsg and LatencyStatProfile, grow excessively large due to a stale profile issue originating from VMware Aria Operations for Networks.
When the Corfu compactor attempts to process these severely bloated tables, it exceeds its allocated memory and crashes. Because the compactor cannot successfully complete its cycle, the database is never compacted, and the /config directory continues to fill up.

Resolution

This issue is resolved in VMware NSX 4.2.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.


If you believe you have encountered this issue and are unable to upgrade, open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.

Additional Information

NSX Manager Data Collection Failure in AON/vRNI Due to Latency Profile Deletion Error