NSX-T Manager UI is not loading, Error: "Some appliance properly components are not functioning properly"

Products

• VMware NSX

Issue/Introduction

 

• NSX-T UI login throws error "error: Some appliance properly components are not functioning"
• NSX-T CLI still shows all services ( https, cluster boot manager, DB etc.) as stable and up across all 3 manager nodes.
• Running df -h as root user show /config is growing in size but is not yet 100% used.

# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/nsx-image 42G 505M 40G 2% /image
/dev/mapper/nsx-config 29G 13G 15G 47% /config >>>>>>>>>>>>>>>>>>
/dev/mapper/nsx-tmp 3.7G 18M 3.5G 1% /tmp
...

• SSH access to the NSX-T Manager node is still possible.
• In the NSX-T manager log /var/log/corfu/corfu-compactor-audit.log we see the following entries:

"ERROR main FrameworkCorfuCompactor - - [XXXXXX comp="nsx-manager" errorCode="MP1" level="ERROR" subcomp="corfu-compactor"] Checkpoint failed for framework data with namespace nsx-manager"

• Just prior to the log entry above, we see the last started check was for table c3ec4373-148f-36f5-xxxx-xxxxxxxxxxx:

INFO main CheckpointWriter - appendCheckpoint: Started checkpoint for c3ec4373-148f-36f5-xxxx-xxxxxxxxxxx at snapshot Token(epoch=4803, sequence=225349
7508)

Environment

VMware NSX-T Data Center

Cause

The table c3ec4373-148f-36f5-xxxx-xxxxxxxxxxx (DynamicGroupMemberComputationResult) grows very large, there is an issue cleaning up the  records in this table and this leads to Corfu being unable to checkpoint the table and causes the Corfu compactor service to go out of memory.

Resolution

This issue is resolved in VMware NSX 3.2.1