MANAGER and HTTPS services are down causing the management cluster to be in degraded state

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are using NSX 4.1 and the NSX-UI is inaccessible.
• In the nsxcli cli as the admin user, when running the command get cluster status, we see the HTTPS and MANAGER service are DOWN:

Group Type: MANAGER
...
    2c231d42-xxxx-xxxx-xxxx-88a7787ef6d0       xxxxxx.xxx                     192.x.x.x    DOWN
...
Group Type: HTTPS
...
    2c231d42-xxxx-xxxx-xxxx-88a7787ef6d0       xxxxxx.xxx                     192.x.x.x    DOWN

• vMotion is blocked as the nsxa agent is down on the transport node(s).
• In /var/log/proton/nsxapi.log in the NSX manager logs you see the following NullPointerException:

Caused by: java.lang.NullPointerException
    at com.vmware.nsx.management.security.firewall.service.FirewallPropertyCache._refresh(FirewallPropertyCache.java:55) ~[?:?]
    at com.vmware.nsx.management.security.firewall.service.FirewallPropertyCache.init(FirewallPropertyCache.java:44) ~[?:?]

• Running the following GET API call from root of an SSH session to the NSX manager you will see an entry as follows, with the key equal to operation:

root@manager1:# curl -k -u 'admin:<password>' -H 'X-NSX-Username:admin' -H 'Content-Type:application/json' -X GET http://127.0.0.1:7440/nsxapi/api/v1/firewall/troubleshoot/properties 

...
    "key" : "ipfix_update_dest_port",
    "value" : "success#2023-05-21T01:00:58.726"
  }, {
    "key" : "operation", 
    "value" : "ipfix_update_dest_port"
  }, {
    "key" : "popularityindextimeunitms",
    "value" : "3600000"
...

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.0.0.1

Cause

This issue is caused when the following internal POST API call is run:

root@manager1:# curl -k -u 'admin:<password>' -H 'X-NSX-Username:admin' -H 'Content-Type:application/json' -X POST http://127.0.0.1:7440/nsxapi/api/v1/firewall/troubleshoot/properties

This API call is for troubleshooting purposes only and should only be run under VMware GSS guidance.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
To work around this issue, contact Broadcom Support and note this Article ID (322438) in the problem description.