NSX Host or Edge Transport Nodes have a status of "Unknown" and Tunnels are "Not Available" or "Validation Errors"

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• NSX Transport Nodes (Host or Edge) status is 'Unknown' or NSX configuration shows 'Validation Errors' in the NSX UI or NSX API 
  
  To get a list of all Transport Nodes and their IDs:
  GET https://<NSX_MANAGER>/api/v1/transport-nodes/

To get the status of a specific Transport Node:
GET https://<NSX_MANAGER>/api/v1/transport-nodes/<tn-id>/status

• Clicking on Transport Node 'Validation Errors' in the NSX UI shows errors similar to the below:
  600: The requested object: <TransportZoneProfile-UUID> could not be found. Object identifiers are case sensitive."
• NSX Transport Nodes tunnel status may show 'Not Available' in the NSX UI.
• No data plane impact has been reported.
• NSX Manager log /var/log/proton/nsxapi.log has a log entry similar to the below:

  "The requested object : TransportZoneProfile/<TransportZoneProfile-UUID> could not be found."

• The above Transport Zone Profile UUID is not present when queried via API:.
  GET https://<NSX_MANAGER>/api/v1/transportzone-profiles?include_system_owned=true

• The above Transport Zone Profile UUID cannot be found in NSX or VC global search.
• The NSX manager log /var/log/search/elasticsearch_index_indexing_slowlog.log has a log similar to example:

  [{"resource_type":"BfdHealthMonitoringProfile","profile_id":"UUID_in_Validation_Error"}]}],"vmk_install_migration":[],"pnics_uninstall_migration":[],"vmk_uninstall_migration":[],"not_ready":false}],"resource_type":"StandardHost]

Environment

VMware NSX-T Data Center

VMware NSX

Cause

The status of the Transport Nodes (TNs) are unknown because the Transport-Zone Profile (TZP) which is being referenced does not exist. The reasons why the referenced Transport-Zone Profile (TZP) is not present in the system can be due to multiple reasons:

• In previous versions of NSX, it was possible in policy mode to update or delete a Transport-Zone Profile (TZP), even if it was in use by a Transport Nodes(TN) or Transport-Node Profile(TNP).
• In some cases, Aria Operations for Networks creates a Transport-Zone Profile (TZP) and when Aria Operations for Networks is removed, Transport-Zone Profile(TZP) are deleted.  Validation has been added in newer versions to avoid this issue.

Resolution

To resolve this issue for Edge nodes, open a support case with Broadcom support.

To resolve this issue for ESXi hosts, follow these steps:

Note: It has been found in one case that removing the Transport Node (Host) from the cluster, waiting for the NSX VIBs to uninstall, then readding the Transport Node to the cluster, resolved the unknown state.

1. Take a new SFTP/FTP based backup and ensure the backup passphrase is known before proceeding.
2. Copy the attached logical-migration.jar file to one of the NSX Manager nodes and place it in the directory /opt/vmware/upgrade-coordinator-tomcat/temp/.
3. Stop the proton service on all three Manager nodes from the root shell:  # service proton stop
   

   Note:  Do not use the "/etc/init.d/proton stop" command, as proton service would start after a few seconds without user intervention)

4. On the NSX Manager node where the jar file was copied, run the following command. This command is a single line command with no line breaks.
5. Note:  You must populate the admin password of the NSX Manager below
   
   # java -Dcorfu-property-file-path=/opt/vmware/upgrade-coordinator-tomcat/conf/ufo-factory.properties -Djava.io.tmpdir=/opt/vmware/upgrade-coordinator-tomcat/temp -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/vmware/upgrade-coordinator-tomcat/conf/log4j2.xml -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/vmware/upgrade-coordinator-tomcat/conf/logging.properties -Dnsx-service-type=nsx-manager -DTransportZoneProfileRectifierInTNAndTNP.userName=admin -DTransportZoneProfileRectifierInTNAndTNP.password='ENTER_ADMIN_PASSWORD_HERE' -DTransportZoneProfileRectifierInTNAndTNP.updateTn=true -DTransportZoneProfileRectifierInTNAndTNP.updateTzp=true -cp /opt/vmware/upgrade-coordinator-tomcat/temp/logical-migration.jar com.vmware.nsx.management.migration.impl.TransportZoneProfileRectifierInTNAndTNP

  Note:  If you download the file multiple times make sure that the file that is provided is as logical-migration.jar, if this is as logical-migration(1).jar it will not run. 
                                                                        
6. Set file ownership: (This step is after issuing the script command in step 5 as the script will be creating new files that will need permissions to write to the upgrade-coordinator.log in order to report "Migration Task Finished")
   # chown uuc:uuc /var/log/upgrade-coordinator/upgrade-coordinator*log* 


7. The procedure is complete once the following text is printed in the upgrade-coordinator log file:
   # grep "Migration task finished" /var/log/upgrade-coordinator/upgrade-coordinator.log
   
   
8. Start proton on all three Manager nodes:
   # service proton start
   
   
9. Execute the following from NSXCLI on all NSX Manager nodes so that CorfuDB and Search Indexes are in sync:
   > start search resync policy
> start search resync manager

Note:  If above command does not resolve the UI issue, execute the following command:
   > start search resync all

    9. Login to the NSX UI and validate that the host status is resolved.

Important:

• In some cases it maybe necessary to detach and re-attach the TNP on impacted cluster to fully resolve the issue.
• For Validation Errors, you may need to select the host, click on Configure NSX > Next > Finish to change the status of the host to Up

In cases where Service VMs are deployed in the cluster the affected host transport nodes are a part of, detaching TNP gives the error:

"Error: Cluster ########-####-####-####-########bed1:domain-c10 has NSX managed service VM deployed or deployment is in progress. Delete these deployment, before deleting TN. (Error code: 26173)". In such a scenario, the alternative to detaching / re-attaching TNP would be to follow the below steps:

1. Place the host into maintenance mode.
2. Run the API "GET https://{{nsx-ip}}/api/v1/transport-nodes/<tn-id>" to get the payload.
3. Using the same payload, run the API "PUT https://{{nsx-ip}}/api/v1/transport-nodes/<tn-id>".
4. Wait for a couple of minutes and check the configuration state of the host.
5. Check the status of the tunnels and in general the status of the host with respect to the manager.
6. Repeat the steps for all the affected hosts.