NSX Host or Edge Transport Nodes have a status of "Unknown" and Tunnels are "Not Available" or "Validation Errors"

Products

VMware NSX

Issue/Introduction

NSX Transport Nodes (Host or Edge) status is Unknown or NSX configuration shows Validation Errors in the NSX UI or NSX API.
To get a list of all Transport Nodes and their IDs:
```
GET https://<NSX_MANAGER>/api/v1/transport-nodes/
```
To get the status of a specific Transport Node:
```
GET https://<NSX_MANAGER>/api/v1/transport-nodes/<tn-id>/status
```

Clicking on Transport Node Validation Errors in the NSX UI shows errors similar to the below:

600: The requested object: <TransportZoneProfile-UUID> could not be found. Object identifiers are case sensitive.

NSX Transport Nodes tunnel status may show Not Available in the NSX UI.

You might see the following validation error when making a change to a Transport Node Profile containing stale references, for example when adding an additional host switch:

Error: Field level validation errors: required property host_switch_spec.host_switches[1].transport_zone_endpoints[0].transport_zone_profile_ids[0].profile_id is missing error code 255

No data plane impact has been reported.

NSX Manager log /var/log/proton/nsxapi.log has a log entry similar to the below:

"The requested object : TransportZoneProfile/<TransportZoneProfile-UUID> could not be found."

The above Transport Zone Profile UUID is not present when queried via API:

GET https://<NSX_MANAGER>/api/v1/transportzone-profiles?include_system_owned=true

The above Transport Zone Profile UUID cannot be found in NSX or VC global search.

The NSX manager log /var/log/search/elasticsearch_index_indexing_slowlog.log has a log similar to example:

[{"resource_type":"BfdHealthMonitoringProfile","profile_id":"UUID_in_Validation_Error"}]}],"vmk_install_migration":[],"pnics_uninstall_migration":[],"vmk_uninstall_migration":[],"not_ready":false}],"resource_type":"StandardHost]

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The status of the Transport Nodes (TNs) are unknown because the Transport Zone Profile (TZP) which is being referenced does not exist. The reasons why the referenced Transport Zone Profile (TZP) is not present in the system can be due to multiple reasons:

In previous versions of NSX, it was possible in policy mode to update or delete a Transport Zone Profile (TZP), even if it was in use by a Transport Nodes(TN) or Transport Node Profile(TNP).
In some cases, Aria Operations for Networks creates a Transport Zone Profile (TZP) and when Aria Operations for Networks is removed, Transport Zone Profile(TZP) are deleted.
Validation has been added in newer versions to avoid this issue.

Resolution

Steps to identify and remove stale TZP references on both Host and Edge Transport Nodes

Take a new SFTP/FTP based backup and ensure the backup passphrase is known before proceeding. Refer to Backing up and restoring the nsx-manager for more information.
Download the attached tar file cleanup_stale_transportzoneprofiles_V3.tar.
Upload the file to an NSX manager with SFTP as the root user into the /tmp directory.
Extract it via CLI as the root user on the NSX manager with the command:
```
tar -xvf cleanup_stale_transportzoneprofiles_V3.tar
```
Stop the proton service on all three Manager nodes from the root shell:
```
service proton stop
```
Note: Do not use the /etc/init.d/proton stop command, as the proton service starts after a few seconds without user intervention.
From the NSX Managers you uploaded and extracted the script to (as root), run the commands in sequence:
- Change to the /tmp directory and run the precheck:
```
cd /tmp
python cleanup_stale_tzp_references.py --precheck
```
- Copy and store the output. Verify the stale references do not exist by selecting the UUID in the output and doing a search in the NSX UI.
- Apply the fix:
```
python cleanup_stale_tzp_references.py --apply
```
- Copy and store the output.
Start the proton service on all three Manager nodes from the root shell:
```
service proton start
```

Note: If no stale entries exist in the --precheck output or the issue persists after applying the fix, please open a case with Broadcom support.

Attachments

cleanup_stale_transportzoneprofiles_V3.tar get_app