Host Transport Node NSX Configuration Stuck at 68% 'Enabling host status in controller'

Products

VMware NSX

Issue/Introduction

Transport nodes that were previously showing completed status for their Configuration State are now showing "Enabling host status in controller" at System > Fabric > Hosts:
The progress becomes stuck at 60% or 68% and never completes.
NSX Manager nodes might have been rebooted. Verify uptime from NSX manager CLI:
nsx-mngr> get uptime
If no reboot was performed, services could have been restarted manually as part of troubleshooting, or automatically if you have open alarms for Application on NSX node has crashed alarm

GET https://<nsx-mgr>/api/v1/transport-nodes/<node-id>/status shows the following output:

"node_uuid" : "####-####-####-####-####",
"node_display_name" : "####",
"status" : "UP",
...
"mgmt_connection_status" : "UP",
"control_connection_status" : {
"status" : "UP", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
"up_count" : 1,
"down_count" : 0,
"degraded_count" : 0
},
...
"node_status" : {
...
"mpa_connectivity_status" : "UP",
"mpa_connectivity_status_details" : "Client is responding to heartbeats",
"lcp_connectivity_status" : "UP",
"lcp_connectivity_status_details" : [ {
...
"status" : "UP"
} ],
"host_node_deployment_status" : "INSTALL_SUCCESSFUL", >>>>>>>>>>>>>>>>>

GET https://<nsx-mgr>/api/v1/transport-nodes/<node-id>/state shows the following output:

	"deployment_progress_state": {
	        "progress": 68,
	        "current_step_title": "Enabling host status in controller"
	    },
	    "state": "in_progress",
	    "details": [
	        {
	            "sub_system_id": "########-####-####-####-############",
	            "sub_system_type": "MaintenanceMode",
	            "state": "success"
	        },
	        {
	            "sub_system_id": "########-####-####-####-############",
	            "sub_system_type": "LogicalSwitchFullSync",
	            "state": "success"
	        },
	        {
	            "sub_system_id": "########-####-####-####-############",
	            "sub_system_type": "HostConfig",
	            "state": "in_progress"
	        }
	    ]
	}

As the failing step is a re-application of the existing config, no changes are actually being made to VMs running on the impacted transport nodes. There should be no data plane impact if the host was previously configured successfully. However, any other services or products that rely on the host state may be impacted as the host state will appear as applying/non-complete to them. In these situations, there may appear messages implying that the Transport Node (TN) config is in-progress.

Environment

VMware NSX 4.1.x
VMware NSX-T Data Center 3.x

Cause

Restarting NSX Manager services or rebooting Manager nodes can lead to a missed AppInit. This prevents the Auto Rectifier service from detecting and resolving the failed transport nodes, leaving the node status in an "Enabling..." state as depicted in the GUI.

Resolution

This issue is resolved in VMware NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

The following API can be run, specifying each impacted ESX transport node to reset the status of that node:

POST https://<NSXT-FQDN>/api/v1/transport-nodes/<transportnode-id>?action=resync_host_config

If the NSX Configuration status is still "Enabling host status in controller" after this, verify if the configuration succeeded for an affected transport node in /var/log/syslog on a manager node:
<timestamp> nsx-mngr NSX 797724 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Got deployment status INSTALL_SUCCESSFUL for node ########-####-###-####-######Perform a rolling restart of the NSX manager nodes to reset the status of the nodes, if you see INSTALL_SUCCESSFUL in syslog.