Alarms in the UI don't show node name which the alarm was raised for

Products

• VMware NSX

Issue/Introduction

• Alarm in the UI is missing node name of the node which this alarm was raised for. 
  
• Also, node_display_name and node_ip_addresses are missing in API GET api/v1/alarms:
   
• Host Transport Nodes which the alarm is triggered for are still present in the system.
• As the alarm doesn't show name of the node, administrator may not be able to easily match the alarm to the actual node.
• This issue was also observed for Host Transport Nodes which were decommissioned from NSX cluster/environment in the past. 

Environment

• VMware NSX-T Data Center
• VMware NSX

Cause

The exact cause of this issue for a Host Transport Node which is still present in the system is yet to be determined.

For a host that has been removed, during Host Transport Node deletion, not all Transport Node related properties are removed from the system. If the system has alarms for such (decommissioned) transport node, it will not be able to match the UUID of the Transport Node with the Transport Node's hostname and IP address (as these properties were already removed when the host was deleted). 

Resolution

To resolve these alarms and to confirm that the Transport Nodes in the alarms do, or do not exist in the system, you can follow the workaround below:

1. Use the API GET api/v1/alarms to dump all open alarms. 
2. Review the output, and identify alarms where fields "node_display_name" and "node_ip_addresses" are empty. 
   If the output is in a text file, you can use the command below to identify all relevant alarms:
   grep "node_display_name\"\:\ \"\"" <file_with_dump_of_api/v1/alarms> -A1 -B14 | grep "\"id\"\|event_type_display_name\|node_id\|node_display_name\|node_ip_addresses" | awk '/"id":/ && NR!=1 { print ""; } { print; }'
3. Identify "node_id" for these alarms. 
4. Use NSX Manager's elastic search to search for the "node_id" found in step 3.
5. Alternatively, you can use the API: 
   GET https://<nsx-manager>/api/v1/transport-nodes
   And search for the UUIDs of Transport nodes in the output.
6. If the node_id (UUID) is not found, you can resolve the alarm through the UI, or using the API:
   POST https://<nsx-manager>/api/v1/alarms/<id>?action=set_status&new_status=RESOLVED

If there are hosts not present in the system, please follow the workaround above.

If there are hosts present in the system with node name missing in the alarm, please open a support case with Broadcom Support and refer to this KB article. Also provide the following:

• NSX Manager support bundles.
• Text of any error messages seen in NSX GUI or command lines pertinent to the investigation (screenshot of alarm(s) in question).
• Output of API call GET https://<nsx-manager>/api/v1/alarms (saved in a text file).

For more information, see Creating and managing Broadcom support cases.

Additional Information

Handling Log Bundles for offline review with Broadcom support:

• Collect Support Bundles for Troubleshooting VMware NSX
• Uploading files to cases on the Broadcom Support Portal
• Creating and managing Broadcom support cases