NSX UI becomes unavailable after upgrade to NSX 4.1.0.x

Products

VMware NSX

Issue/Introduction

NSX 4.1.0.x
Environment has been upgraded or in the process of being upgraded. NSX is not greenfield deployment

NSX UI is down and users cannot login

Some appliance components are not functioning properly.
Component health: MANAGER:UNKNOWN, SEARCH:UNKNOWN, NODE_MGMT:UP, UI:UP.
Error code: 101

All 3 NSX Managers are running the same version, validate this via ssh as root user by checking /etc/nsx_issue

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted

Every 10 seconds, one of the NSX Managers reports that Corfu compaction has been disabled

/config partition usage consistently grows over time. In a steady state, /config usage should typically be less than 5% e.g.

# df -h /config
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/nsx-config 29G 4.4G 24G 17% /config

May be indications of crash hprof files for cbm or proton generated e.g.

# ls -l /image/core/
-rw------- 1 nsx-cbm nsx-cbm 257201917 May 18 15:49 cbm_oom.hprof

Environment

VMware NSX 4.1.0.x

Cause

During an upgrade to NSX 4.1.0.x, the compactor service is disabled by design. This issue occurs when the upgrade silently fails before re-enabling the compactor.

Resolution

This issue is resolved in NSX 4.1.1, available at Broadcom downloads.

Workaround:

1) Reconfirm that all 3 NSX Managers are running the same version before proceeding to step 2. There is potential for data loss if all Managers are not running the same version. ssh as root user to all 3 Managers

#cat /etc/nsx_issue
version: 4.1.0.0.0.21332677
node-type: nsx-manager nsx-policy-manager nsx-controller
build-type: release
export-type: unrestricted

2) Confirm all NSX Managers have completed the upgrade:

get upgrade progress-status

The NSX Manager should report: Upgrade is not in progress

If Upgrade is in progress, complete the upgrade process or contact GSS for assistance with completing the upgrade

3) To resolve the issue, run the following command on 1 Manager as root user, it can be on any of the Managers

#/usr/share/corfu/scripts/compactor_runner.py --port 9000 --compactorConfig /usr/share/corfu/conf/corfu-compactor-config.yml --enableCompaction=True

4) This will re-enable Corfu compaction again, which will run automatically every 15 minutes.

It may take many iterations of compaction running before a reduction in /config size is noticed from 'df -h' output, meaning it may take potentially hours.
Ensure that the "Compaction has been disabled" log message matched in Symptoms is no longer observed.