NSX host transport node stuck at 80% Enabling Host Status in Controller after install/re-install.

Products

• VMware NSX

Issue/Introduction

When configuring an ESXi host as an NSX Transport Node, the deployment progress can become stuck at 80% (Enabling host status in controller) within the NSX Manager UI.  

• Host may have previously shown as successful for NSX until a reboot or MP agent restart which allowed VMs to be deployed to it using NSX elements.
• Data plane network connectivity fails for Virtual Machines residing on the impacted host for overlay connected VMs.
• Other TN and Edges within the environment may show tunnel down states to the problematic host, while the problematic TN has a reduced number of tunnels but 100% Up.
• TN previously was configured with NSX within the environment with a VDS that no longer exists in the environment.
• NSX Manager logs show (var/log/syslog)
  
  ERROR L2HostConfigTaskExecutor1 ProjectVpcVCSyncerUtils 6303 SWITCHING [nsx@6876 comp="nsx-manager" errorCode="MP9538" level="ERROR" subcomp="manager"] Could not find the HostSwitch [<VDS UUID>] of type VDS
  ERROR L2HostConfigTaskExecutor1 TransportNodeAsyncRequestHandler 6303 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP9538" level="ERROR" subcomp="manager"] TransportNodeConfigTask on host [<Problem TN UUID>] failed. Abort all unprocessed stages.
  
  
• Running the command "nsxcli -c get gateways" on the ESXi host shows 0 Virtual Distributed Router (VDR) Logical Interfaces (LIFs) present on the production Virtual Distributed Switch (VDS) while VMs are present that are connected to a segment with a DR as its gateway:  
  
  

DvsName                  VdrPort              NumLifs   DRvmac
---                  -------              -------   ------
exampleDVS           vdrPort                   0         02:50:56:56:44:52

• Querying the transport node deployment state via the NSX Manager API returns a failure pointing to a missing or unrecognized HostSwitch UUID, it may also show success at the same time.

curl -k -u 'admin' -X GET "https://localhost/api/v1/transport-nodes/<TN-UUID>/state
{
  "transport_node_id" : "<TNUUID>",
  "maintenance_mode_state" : "DISABLED",
  "node_deployment_state" : {
    "state" : "success",
    "details" : [ ]
  },
  "deployment_progress_state" : {
    "progress" : 80,
    "current_step_title" : "Enabling host status in controller" <<< Same state as GUI.
  },
  "state" : "in_progress",
  "details" : [ {
    "sub_system_id" : "<TN-UUID>",
    "sub_system_type" : "Start",
    "state" : "failed",
    "failure_message" : "Cannot find the HostSwitch [<Non-Existent-DVS-UUID>] of type VDS.",
    "failure_code" : 9538
  } ]
}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This issue occurs because the transport node previously was configured within NSX with a binding to a VDS. The host was subsequently removed but not fully completed (as per Best Practice for Decommissioning an NSX-Prepared ESXi Host to Avoid Stale Entries) and a partial record binding the VDS entry remains. During host re-preparation, the host sync state fails during the attempted clean-up of what it previously had configured (a now non existent VDS) leaving NSX with a partial new and partial old config. The host then attempts to bind VDR LIFs to the old non existent VDS as it was in a process of updating/rollback when it failed on the clean-up. This leaves an inconsistent config state and the host unable to complete its configuration.

Resolution

This issue is resolved in VCF 9.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

To workaround this behaviour, you must fully purge the invalid state configuration records from NSX by fully deleting the TN record from NSX by following Admin guide - Uninstall NSX from a managed ESXI host. The removal may also fail and get stuck at 20% with a status of disabling host status in controller.  If so then please follow the workaround steps provided in the KB  Installing NSX on Host fails with "already exists" error or upgrade fails with "Failed to get Host status"

If after actioning both KBs the TN remains, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.