NSX UI Outage /Unresponsive with 'Unable to create CompactorLeaderServices object' Error in NSX 4.1.0

Products

• VMware NSX
• VMware NSX

Issue/Introduction

Unresponsive NSX UI due to growing /config partition usage and caused CompactorLeaderServices Object Creation Failure

• NSX Version is 4.1.0

• /config partition usage consistently grows over time. In a steady state, /config usage should typically be less than 5% . For instance

  # df -h /config
  Filesystem                        Size  Used Avail Use% Mounted on
  /dev/mapper/nsx-config   29G  4.4G   24G  17% /config

   

• Crash hprof files for cbm or proton may be generated. For instance

      # ls -l /image/core/
      -rw------- 1 nsx-cbm nsx-cbm 257201917 May 18 15:49 cbm_oom.hprof

• The NSX UI might become unresponsive.
• To identify this issue, the user should observe the message "Unable to create CompactorLeaderServices object" in the /var/log/corfu/corfu-compactor-leader.log file on one of the NSX Managers. An example error message is as follows:

<timestamp>.496Z | ERROR | initializationTaskThread | compactor-leader | Unable to create CompactorLeaderServices object. Will retry on next attempt. Exception:
org.corfudb.runtime.exceptions.UnreachableClusterException: Runtime stalled. Invoking systemDownHandler after 60 unsuccessful tries.
        at org.corfudb.infrastructure.ManagementServer.lambda$new$0(ManagementServer.java:99)
        at org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:176)
        at org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:61)
        at org.corfudb.runtime.view.SequencerView.lambda$query$2(SequencerView.java:40)
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:57)
        at org.corfudb.common.metrics.micrometer.MicroMeterUtils.lambda$time$6(MicroMeterUtils.java:121)
        at java.util.Optional.map(Optional.java:215)

Environment

VMware NSX 4.1.0

Cause

This issue occurs due to an unstable underlying infrastructure. 

Resolution

The issue has been resolved in NSX 4.1.0.2 and higher.

Workaround:

To mitigate the issue temporarily, restart the Corfu server.

Additional Information

• This issue will not impact any other version other than 4.1.0.
  • NSX-T 4.1.0.2 has the fix.
  • corfu-compactor-leader concept was introduced in 4.1.0, this issue will not impact any prior version