NSX UI becomes unavailable on NSX 4.1.0 or 4.1.0.2

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX 4.1.0 or 4.1.0.2
• Environment has been upgraded and is not greenfield
• NSX UI is down and users cannot login
• All 3 NSX Managers are running the same version, validate this via ssh as root user by checking /etc/nsx_issue

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted

• Every 10 seconds, one of the NSX Managers reports that Corfu compaction has been disabled

/var/log/corfu/corfu-compactor-leader.log
    2023-05-22T17:22:59.728Z | WARN | Cmpt-9000-chkpter | compactor-leader | Compaction has been disabled
    2023-05-22T17:23:09.732Z | WARN | Cmpt-9000-chkpter | compactor-leader | Compaction has been disabled

• /config partition usage consistently grows over time. In a steady state, /config usage should typically be less than 5% e.g.

# df -h /config
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/nsx-config   29G  4.4G   24G  17% /config

• Crash hprof files for cbm or proton may be generated e.g.

# ls -l /image/core/ 
-rw------- 1 nsx-cbm nsx-cbm 257201917 May 18 15:49 cbm_oom.hpro

Environment

VMware NSX 4.1.0

Cause

During an upgrade to NSX 4.1.0 or 4.1.0.2, the compactor service is disabled by design. This issue occurs when the upgrade silently fails before re-enabling the compactor.

Resolution

This issue is resolved in NSX 4.1.1, available to download from the Customer Connect portal.

Workaround:

1) Reconfirm that all 3 NSX Managers are running the same version, either 4.1.0 or 4.1.0.2 before proceeding to step 2. There is potential for data loss if all Managers are not running the same version. ssh as root user to all 3 Managers

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted
 

2) To resolve the issue, run the following command on 1 Manager as root user, it can be on any of the Managers

#/usr/share/corfu/scripts/compactor_runner.py --port 9000 --compactorConfig /usr/share/corfu/conf/corfu-compactor-config.yml --enableCompaction=True

3) This may take some time to complete, potentially hours.
Monitor the /config partition usage with the df command and confirm it is reducing in size.
Ensure that the "Compaction has been disabled" log message matched in Symptoms is no longer observed.