NSX UI becomes unavailable on NSX 4.1.0 to 4.1.2.4

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX 4.1.0 - 4.1.2.4
• Environment has been upgraded or in the process of being upgraded.  NSX is not greenfield deployment
• NSX UI is down and users cannot login
• All 3 NSX Managers are running the same version, validate this via ssh as root user by checking /etc/nsx_issue

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted

• Every 10 seconds, one of the NSX Managers reports that Corfu compaction has been disabled

/var/log/corfu/corfu-compactor-leader.log
    2023-05-22T17:22:59.728Z | WARN | Cmpt-9000-chkpter | compactor-leader | Compaction has been disabled
    2023-05-22T17:23:09.732Z | WARN | Cmpt-9000-chkpter | compactor-leader | Compaction has been disabled

• /config partition usage consistently grows over time. In a steady state, /config usage should typically be less than 5% e.g.

# df -h /config
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/nsx-config   29G  4.4G   24G  17% /config

• May be indications of crash hprof files for cbm or proton generated e.g.

# ls -l /image/core/ 
-rw------- 1 nsx-cbm nsx-cbm 257201917 May 18 15:49 cbm_oom.hpro

Environment

VMware NSX 4.1.0 - 4.1.2.4

Cause

During an upgrade to NSX 4.1.0, 4.1.0.2 or 4.1.2.4, the compactor service is disabled by design. This issue occurs when the upgrade silently fails before re-enabling the compactor.

Resolution

Workaround:

1) Reconfirm that all 3 NSX Managers are running the same version, either 4.1.0, 4.1.0.2, or 4.1.2.4 before proceeding to step 2. There is potential for data loss if all Managers are not running the same version. ssh as root user to all 3 Managers

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted

 

         2) Confirm all NSX Managers have completed the upgrade:

                  get upgrade progress-status

                         The NSX Manager should report: Upgrade is not in progress

                         If Upgrade is in progress, complete the upgrade process or contact GSS for assistance with completing the upgrade

 

3) To resolve the issue, run the following command on 1 Manager as root user, it can be on any of the Managers

#/usr/share/corfu/scripts/compactor_runner.py --port 9000 --compactorConfig /usr/share/corfu/conf/corfu-compactor-config.yml --enableCompaction=True

4) This may take some time to complete, potentially hours.
Monitor the /config partition usage with the df -h command from root CLI and confirm it is reducing in size.
Ensure that the "Compaction has been disabled" log message matched in Symptoms is no longer observed.