"Some appliance components are not functioning properly - Error code: 101" Error displayed when attempting to log in to NSX manager with no active NSX backup

Products

• VMware NSX

Issue/Introduction

• A user is unable to access the NSX GUI and cannot log in.  The user may not be able to access all/one of the NSX Manager nodes via the web.
• The following or similar alerts are displayed on the browser when trying to access NSX Manager UI or NSX Global Manager UI:
• Some appliance components are not functioning properly.
  Component health: SEARCH:UNKNOWN, MANAGER:UNKNOWN, NODE_MGMT:UP, UI:UP.
Error code: 101
• The following error is returned when the 'get cluster status' command is run on the manager CLI:
  <Manager Name> get cluster status
% An error occurred while getting the cluster status
• The corfu compactor service has not run for a long time, which typically runs every 15 minutes, to confirm if they have completed:
  Login to NSX Manager as root user and review the /var/log/corfu/corfu-compactor-audit.log to see the last messages. Review the timestamp and when the corfu compactor last finished:
  <Timestamp>  | INFO  |              Cmpt-chkpter-9000 |    o.c.c.CompactorCheckpointer | Exiting CorfuStoreCompactor
<Timestamp>  INFO Runner - Finished running corfu compactor tool.
• No valid or recent file-based backups exist on the external SFTP server.
• Inability to download or generate support bundles/logs from the NSX Manager due to disk space exhaustion or service failure.

Environment

VMware NSX-T 4.x

Cause

If there is no corfu compactor leader change for a long time, the corfu compactor leader may become unresponsive and not trigger compaction cycles, unless Corfu server is restarted. This leads to CBM service running out of memory and crashing.

Resolution

Perform a rolling reboot of the NSX Manager nodes as outlined in the KB article above.

Note: If the /config directory does not fall to 1% after the rolling reboot, and if there are no active backups to restore the NSX, then open a support case with Broadcom Support and reference this KB article.

Additional Information

Refer to this article for more insights on JDK bug impacting corfu compactor.