NSX Manager appliance status / service status is not UP

Products

• VMware NSX

Issue/Introduction

NSX Manager Appliance node status / Service status is in DOWN/DEGRADED/UNKNOWN state.

Impact:
    - In a clustered manager setup, high availability and performance will be impacted.
    - In a singleton manager setup, management plane and control plane will be affected.

Environment

• VMware NSX
• VCF 9.0 and above 

Resolution

1. If all the services of a node are in UNKNOWN/DOWN state then
     a. Please ensure that the manager appliance node VM is powered on and running.
     b. If VM is powered on, then this implies monitoring services could be down.
        Please check the runtime status of sha, appliance proxy and monitoring services using nsx cli "get service <service-name>" to check the service status.
        If any of these services are not running, please start these services using command "start service <service-name>"

2. If some services are in DOWN/DEGRADED/UNKNOWN state, please refer to error message corresponding to node status for details on which services are in DOWN/DEGRADED/UNKNOWN state.
Perform the below steps for all the services that are in DOWN/UNKNOWN/DEGRADED state. Use the service name mapping provided below while referring to the services in CLI.
     a. If either datastore/datastore_nonconfig services are not in UP state, please refer to KB article Corfu/Corfu-nonconfig status is not up
         If search service is not in UP state, please refer to KB article NSX Manager Search service status is not UP
         For other services follow below steps (b) to (e)
     b. Invoke the CLI command “get service <service-name>" using service name to check for the service status.
     c. 
Check /var/log/nsx-syslog.log and /var/log/syslog.log to see if there are errors reported corresponding to the service.
     d. If service is not running, invoke "start service <service-name>" to start the service. If service is already running, invoke "restart service <service-name>" to restart the service. Monitor the logs in /var/log/syslog.log and /var/log/nsx-syslog.log for any errors.
     e. Wait for 5 minutes for the appliance node/service status to be updated in UI.

If further assistance required, please open a case with Broadcom Support using Creating and managing Broadcom support cases

Additional Information

Service name mapping between UI and CLI are as follows

      CLUSTER_BOOT_MANAGER : cluster_manager
      CONTROLLER : controller
      CM-INVENTORY : cm-inventory
      DATASTORE : datastore
      DATASTORE_NONCONFIG : datastore_nonconfig
      AUTH : auth
      MANAGER : manager
      MESSAGING_MANAGER : messaging-manager
      MONITORING : monitoring
      SEARCH : search
      SITE_MANAGER : site_manager
      APPLIANCE_PROXY : appliance proxy
      SHA : sha