NSX manager UI is not showing alarms after rolling reboot of NSX manager nodes Post hitting the JDK issue.

Products

• VMware NSX

Issue/Introduction

• After performing a rolling reboot on NSX Manager nodes (often as a workaround for a JDK issue), alarms are no longer visible in the NSX Manager UI
  
• The cm-inventory service fails to start automatically following the reboot.
  Running the command "get services" on the NSX Manager CLI shows the following state: 
  

  • NSX > get services
Service name:                          applianceproxy
Service state:                           running

    Service name:                          async_replicator
Service state:                           running
Logging level:                           info

    Service name:                          auth
Service state:                           running
Logging level:                           info

    Service name:                          cluster_manager
Service state:                           running

    Service name:                          cm-inventory
Service state:                           Stopped 

Environment

• VMware NSX 4.2.0.x
• VMware NSX 4.2.1.3 and earlier

Cause

• The rolling reboot can cause the cm-inventory service to fail during initialization.
• Logs located at /var/log/cm-inventory/cm-inventory may show the following errors indicating a connection failure with the Appliance Proxy Hub (APH):
  INFO pool-33-thread-1 AbstractView 4563 layoutHelper: Retried 0 times, SystemDownHandlerTriggerLimit = 60
INFO pool-33-thread-1 AphConnProviderService 4563 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Forwarding connection with APH went down.
INFO pool-33-thread-1 StubManager 4563 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Forwarding connection with APH went down.
at org.corfudb.runtime.NodeRouterPool.shutdown(NodeRouterPool.java:56) ~[runtime-4.1.20230509211150.7973.1.jar:?]
at org.corfudb.runtime.CorfuRuntime.shutdown(CorfuRuntime.java:1017) ~[runtime-4.1.20230509211150.7973.1.jar:?]
INFO pool-33-thread-1 AbstractView 4563 layoutHelper: Retried 0 times, SystemDownHandlerTriggerLimit = 60
ERROR nsx-rpc:RPC_PROXY_CONN_PROVIDER:keepalive-0 rejectedExecution 4563 Failed to submit a listener notification task. Event loop shut down?

Resolution

To restore UI alarms, manually restart the cm-inventory service on the affected NSX Manager node(s).

Steps:

• Identify which NSX Manager node is reporting the cm-inventory service as **Stopped**.
• Ensure you have root access to the NSX Manager node.
• Restart the service by executing the following command:
   /etc/init.d/cm-inventory restart
• Verify the service state from the NSX CLI:
   NSX>get services (Confirm the cm-inventory service state is now running)

Additional Information

• For JDK related issue please refer to the kb : https://knowledge.broadcom.com/external/article/390592