MANAGER and HTTPS services are down causing the management cluster to be in degraded state

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• The NSX-UI is inaccessible.
• In the nsxcli cli as the admin user, when running the command get cluster status, you see the HTTPS and MANAGER service are DOWN:

Group Type: MANAGER
...
    2c231d42-####-####-####-88a7787ef6d0       ########                     192.#.#.#    DOWN
...
Group Type: HTTPS
...
    2c231d42-####-####-####-88a7787ef6d0       ########                     192.#.#.#    DOWN

• vMotion is blocked as the nsxa agent is down on the transport node(s).
• In /var/log/proton/nsxapi.log in the NSX manager logs you see the following NullPointerException:

Caused by: java.lang.NullPointerException
    at com.vmware.nsx.management.security.firewall.service.FirewallPropertyCache._refresh(FirewallPropertyCache.java:55) ~[?:?]
    at com.vmware.nsx.management.security.firewall.service.FirewallPropertyCache.init(FirewallPropertyCache.java:44) ~[?:?]

• Running the following GET API call from root of an SSH session to the NSX manager you will see an entry as follows, with the key equal to operation:

root@manager1:# curl -k -u 'admin:<password>' -H 'X-NSX-Username:admin' -H 'Content-Type:application/json' -X GET http://127.0.0.1:7440/nsxapi/api/v1/firewall/troubleshoot/properties 

...
    "key" : "ipfix_update_dest_port",
    "value" : "success#2023-05-21T01:00:58.726"
  }, {
    "key" : "operation", 
    "value" : "ipfix_update_dest_port"
  }, {
    "key" : "popularityindextimeunitms",
    "value" : "3600000"
...

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.0.0.1

Cause

This issue is caused when the following internal POST API call is run:

curl -k -u 'admin:<password>' -H 'X-NSX-Username:admin' -H 'Content-Type:application/json' -X POST http://127.0.0.1:7440/nsxapi/api/v1/firewall/troubleshoot/properties

This API call is for troubleshooting purposes only and should only be run under VMware GSS guidance.

Resolution

This issue is resolved in VMware NSX 4.1.1.0
This issue is resolved in VMware NSX 4.1.2.0

Workaround:
To work around this issue, contact Broadcom Support and note this Article ID (322438) in the problem description.