ESXi Host NSX Preparation Stuck at "Enabling Host Status in Controller"

Products

VMware NSX VMware NSX-T Data Center

Issue/Introduction

- The ESXi host NSX preparation status is stuck at 68% or 80% "Enabling Host Status in Controller.

- Under NSX MGR UI > System > Fabric > Clusters > Host Details, the Configuration State is stuck at "Enabling host status in controller".

- Despite the host preparation status, the VMs running on the impacted hosts and using NSX networking will continue to perform normally with no issues.

- This issue usually occurs after:

A recent reboot of the NSX Manager.
A recent NSX upgrade.
NSX configuration changes requiring a host resynchronization.
Network disruptions causing the ESXi host to disconnect from the NSX Controller.

- In some cases, the logs may show failure codes, such as 8804 or 8814, indicating the root cause. The specific error code may vary based on the underlying issue. Below are some logs examples:

GET https://<nsx manager>/api/v1/transport-nodes/<transport-node-id>/state:

{ "details": [ { "failure_code": 8804, "failure_message": "Reconfiguring after last failure: LogicalSwitch full-sync: Status is in-progress on '1' transport nodes.", "state": "in_progress", "sub_system_id": "########-####-####-####-############", "sub_system_type": "Host" } ], "failure_code": 8804, "failure_message": "Host configuration failed. Number of retries : 3005. Next retry attempt will be between 2023-Sep-26 13.30.15 PM and 2023-Sep-26 13.33.15 PM (UTC).", "maintenance_mode_state": "DISABLED", "node_deployment_state": { "details": [], "state": "success" }, "state": "in_progress", "transport_node_id": "########-####-####-####-############"       
},

nsxapi.log: (Log full path : /var/log/proton/nsxapi.log)

2022-02-09T12:29:32.341Z INFO L2HostConfigTaskExecutor2 HostTNDeploymentProgressServiceImpl 11278 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TN Operation: UPDATE. Host: ########-####-####-####-############ current progress percentage: 80, stateDescription: deployment.progress.tn.create.waiting_for_host_config_reply

2022-02-09T12:29:32.365Z INFO L2HostConfigTaskExecutor2 TransportNodeAsyncServiceImpl 11278 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TN TransportNode/########-####-####-####-############ realization state : IN_PROGRESS, convergence state: IN_PROGRESS, realization message : Status is in-progress on '1' transport nodes.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The host preparation process may get stuck at different percentages (e.g., 68%, 80%). When it stalls at the "Enabling host status in controller" stage, this typically indicates a synchronization issue between the host's data plane and the NSX Manager control plane.

The issue could be a result of:

- Network connectivity issues between the host and the NSX Manager.
- Resource contention on the NSX Manager if multiple hosts were undergoing redeployment simultaneously.
- A hung process either on the ESXi host or the NSX Manager side.

Resolution

To avoid running into the issue, ensure reliable network connectivity between the host and the NSX Manager and make sure there is no resource contention by proper capacity planning and gradual preparation for the host clusters. The below actions could help to resolve the issue:

Resync the impacted ESXi Host:
- In most cases, rebooting the affected ESXi host forces a resync and resolves the issue.
Reboot the NSX Manager (if necessary):
- If the issue persists after rebooting the host, reboot the NSX Manager to reestablish communication between components.
Put the ESXi Host in NSX Maintenance Mode, then exit NSX Maintenance Mode
Uninstall NSX on the affected host and reinstall:
- If the issue occurred after the recent NSX upgrade, NSX reinstall process might resolve the issue if above steps didn't resolve.
  Follow this techdoc to force uninstall NSX on the host Uninstall NSX from a Managed Host in a vSphere Cluster

If the issue persists, please reach out to the Broadcom Support team to assist.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following: Reference KB "Creating and managing Broadcom support cases"

NSX Manager support bundles
ESXi host support bundles for hosts that are failing to configure as transport nodes.
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support