In NSX-T 3.1.3.x the MANAGER, CONTROLLER and HTTPS status DOWN at cluster level on all nodes.

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are using NSX-T 3.1.3.x.
• The MANAGER, CONTROLLER and HTTPS status are DOWN at cluster level on all three nodes.
• Logs similar to the below will be present in var/log/proton/nsxapi.log

2022-02-24T12:44:54.837Z ERROR coordinationEventsProcessor-1 ApplicationContextManager 14456 - [nsx@6876 comp="nsx-manager" errorCode="MP2112" level="ERROR" subcomp="manager"] Failed to start management application.
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'switchingConfiguration': Unsatisfied dependency expressed through method 'setLogicalSwitchStateAutoRectifier' parameter 0; nested exception
is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'logicalSwitchStateAutoRectifier': Unsatisfied dependency expressed through field 'lsStateService'

 
 

• Null pointer exceptions will be observer in var/log/proton/nsxapi.log with the cause appearing to be caused by NfwConfigServiceImpl

caused by: java.lang.NullPointerException at com.vmware.nsx.management.nfwconfig.service.NfwConfigServiceImpl.getUriFilters(NfwConfigServiceImpl.java:307) ~[libnfw-config.jar:?]

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

This issue is caused when a watcher in NfwNotificationIdWatcherId config fails a deletion and doesn't remove some of the mappings from NfwNotificationIdWatcherId config table.
Subsequently, when we try to find that watcher we don't find the entry in the NfwNotificationWatcherConfig table.

Resolution

Currently there is no resolution to this issue. 

Workaround:
In the event that an upgrade is not possible, please raise a support request with Vmware quoting this article.