NSX Host or Edge Transport Nodes have a status of "Unknown" and Tunnels are "Not Available" or "Validation Errors"

Products

• VMware NSX

Issue/Introduction

 

• NSX Transport Nodes (Host or Edge) status is 'Unknown' or NSX configuration shows 'Validation Errors' in the NSX UI or NSX API 
  
  To get a list of all Transport Nodes and their IDs:
  

  GET https://<NSX_MANAGER>/api/v1/transport-nodes/

  
  To get the status of a specific Transport Node:
  

  GET https://<NSX_MANAGER>/api/v1/transport-nodes/<tn-id>/status

• Clicking on Transport Node 'Validation Errors' in the NSX UI shows errors similar to the below:
  

  600: The requested object: <TransportZoneProfile-UUID> could not be found. Object identifiers are case sensitive."

• NSX Transport Nodes tunnel status may show 'Not Available' in the NSX UI.
  
  
  
  
  
• No data plane impact has been reported.
• NSX Manager log /var/log/proton/nsxapi.log has a log entry similar to the below:
  

  "The requested object : TransportZoneProfile/<TransportZoneProfile-UUID> could not be found."

• The above Transport Zone Profile UUID is not present when queried via API:
  

  GET https://<NSX_MANAGER>/api/v1/transportzone-profiles?include_system_owned=true

• The above Transport Zone Profile UUID cannot be found in NSX or VC global search.
• The NSX manager log /var/log/search/elasticsearch_index_indexing_slowlog.log has a log similar to example:
  

  [{"resource_type":"BfdHealthMonitoringProfile","profile_id":"UUID_in_Validation_Error"}]}],"vmk_install_migration":[],"pnics_uninstall_migration":[],"vmk_uninstall_migration":[],"not_ready":false}],"resource_type":"StandardHost]

Environment

• VMware NSX-T Data Center
• VMware NSX

Cause

The status of the Transport Nodes (TNs) are unknown because the Transport Zone Profile (TZP) which is being referenced does not exist. The reasons why the referenced Transport Zone Profile (TZP) is not present in the system can be due to multiple reasons:

• In previous versions of NSX, it was possible in policy mode to update or delete a Transport Zone Profile (TZP), even if it was in use by a Transport Nodes(TN) or Transport Node Profile(TNP).
• In some cases, Aria Operations for Networks creates a Transport Zone Profile (TZP) and when Aria Operations for Networks is removed, Transport Zone Profile(TZP) are deleted.
• Validation has been added in newer versions to avoid this issue.

Resolution

Steps to identify and remove stale TZP references on both Host and Edge Transport Nodes

1. Take a new SFTP/FTP based backup and ensure the backup passphrase is known before proceeding. Refer to Backing up and restoring the nsx-manager for more information.
2. Create a temporary folder, say /root/script-based-cleanup/. Copy the attached 'cleanup_stale_tzp_references_V3.py' file to one of the NSX Manager nodes and place it in the directory /root/script-based-cleanup/.
3. Stop the proton service on all three Manager nodes from the root shell:
   1. $ service proton stop
NOTE: Do not use the "/etc/init.d/proton stop" command, as proton service would start after a few seconds without user intervention.
4. From one of the NSX Managers (as root), run the commands in sequence:
   1. Run 'precheck' command as below. 
      $ python cleanup_stale_tzp_references_V3.py --precheck
   2. Verify the stale references do not exist by selecting the UUID in the output and doing a search in the NSX UI
   3. Apply the fix
$ python cleanup_stale_tzp_references_V3.py --apply
5. Start the proton service on all three Manager nodes from the root shell:
   1. service proton start

 

If no stale entries exist in the --precheck output or the issue persists after applying the fix, please open a case with Broadcom support.